2024-03-25T16:15:57.041 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:15:57.041 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --owner scheduled_amathuri@teuthology --verbose --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621263 --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621263/orig.config.yaml 2024-03-25T16:15:57.086 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:15:57.175 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621263 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621263' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOsESA1elaXn55kueK3fg/Dl/CEXysJyvHdNcX1qTpJ6zrBpeeQqkkwPdMgV3RBpENoBbyHZLR0nrbBUrKKPduo= smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBh9+NaMT+DLhes+BYo8wW/byWZV2VoUHyX7ijMCUMoyWHexkOSxRdg1lbBDy8P4D9z1Hprw2BYNCbuEm1Uygsc= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:15:57.175 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:15:57.176 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:15:57.176 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:15:57.183 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:15:57.210 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:15:57.210 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:15:57.210 INFO:teuthology.packaging:ref: None 2024-03-25T16:15:57.210 INFO:teuthology.packaging:tag: None 2024-03-25T16:15:57.210 INFO:teuthology.packaging:branch: main 2024-03-25T16:15:57.211 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:15:57.211 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:15:57.581 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:15:57.583 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:15:57.592 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:15:57.592 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:15:57.613 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:15:57.630 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:15:57.636 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:15:57.661 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621263', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:10:59.941756', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOsESA1elaXn55kueK3fg/Dl/CEXysJyvHdNcX1qTpJ6zrBpeeQqkkwPdMgV3RBpENoBbyHZLR0nrbBUrKKPduo='} 2024-03-25T16:15:57.685 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621263', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:10:59.944183', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBh9+NaMT+DLhes+BYo8wW/byWZV2VoUHyX7ijMCUMoyWHexkOSxRdg1lbBDy8P4D9z1Hprw2BYNCbuEm1Uygsc='} 2024-03-25T16:15:57.685 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:15:57.692 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:15:57.692 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:15:57.692 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:15:57.767 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7efdca59d0d0>, signals=[15]) 2024-03-25T16:15:57.767 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:15:57.775 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:15:57.775 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-03-25T16:15:57.777 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:15:57.887 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-03-25T16:15:57.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:15:57.977 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:15:57.984 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-03-25T16:15:58.001 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-03-25T16:15:58.001 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-03-25T16:15:58.060 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-03-25T16:15:58.060 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-03-25T16:15:58.060 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-03-25T16:15:58.060 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:15:58.060 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-03-25T16:15:58.060 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:15:58.061 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:15:58.061 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-03-25T16:15:58.061 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:15:58.061 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:15:58.061 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:15:58.061 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:15:58.061 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:15:58.061 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:15:58.062 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-03-25T16:15:58.088 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-03-25T16:15:58.105 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-03-25T16:15:58.105 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-03-25T16:15:58.159 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2024-03-25T16:15:58.159 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="9" 2024-03-25T16:15:58.160 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2024-03-25T16:15:58.160 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:15:58.160 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="9" 2024-03-25T16:15:58.160 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:15:58.160 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:15:58.160 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2024-03-25T16:15:58.160 INFO:teuthology.orchestra.run.smithi086.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:15:58.160 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:15:58.160 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:15:58.160 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:15:58.160 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:15:58.160 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:15:58.161 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-03-25T16:15:58.179 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:15:58.198 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:15:58.204 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:15:58.205 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:15:58.207 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:15:58.222 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:15:58.228 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:15:58.228 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:15:58.268 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:15:58.314 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:15:58.666 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:15:58.686 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:15:58.686 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:15:58.686 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:15:58.687 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:15:58.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:15:58.703 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-03-25T16:15:58.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:15:58.704 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-03-25T16:15:58.759 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:15:58.760 INFO:teuthology.task.kernel:Running kernel on smithi086: 5.14.0-361.el9.x86_64 2024-03-25T16:15:58.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2024-03-25T16:15:58.766 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:15:58.766 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.14.0-361.el9.x86_64 2024-03-25T16:15:58.766 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2024-03-25T16:16:00.952 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-25T16:16:01.172 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - AppStream 8.6 MB/s | 6.7 MB 00:00 2024-03-25T16:16:02.460 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-25T16:16:02.814 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-25T16:16:03.217 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-25T16:16:04.396 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.0 MB 00:01 2024-03-25T16:16:04.919 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-03-25T16:16:09.413 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 7.3 MB/s | 6.1 MB 00:00 2024-03-25T16:16:11.062 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-25T16:16:11.163 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 3.3 MB/s | 19 MB 00:05 2024-03-25T16:16:12.296 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-25T16:16:16.614 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.1 MB 00:02 2024-03-25T16:16:17.346 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-25T16:16:18.206 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-25T16:16:19.511 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:16:19.512 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:16:19.614 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:16:19.621 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:16:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-03-25T16:16:19.621 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:16:19.621 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:16:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:16:19.621 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-25T16:16:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:16:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:16:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:16:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:16:19.622 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:16:19.622 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:16:19.622 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-03-25T16:16:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:16:19.622 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 76 M 2024-03-25T16:16:19.623 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 123 M 2024-03-25T16:16:19.623 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:16:19.791 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-25T16:16:20.064 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 277 kB/s | 44 kB 00:00 2024-03-25T16:16:21.540 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 9.1 MB/s | 15 MB 00:01 2024-03-25T16:16:22.439 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 12 MB/s | 27 MB 00:02 2024-03-25T16:16:22.815 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 11 MB/s | 33 MB 00:02 2024-03-25T16:16:22.815 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:16:22.815 INFO:teuthology.orchestra.run.smithi064.stdout:Total 24 MB/s | 76 MB 00:03 2024-03-25T16:16:23.511 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:16:23.938 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:16:23.938 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:16:24.724 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-25T16:16:24.952 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:16:24.952 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:16:26.764 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:16:26.765 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:16:26.861 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:16:26.867 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:16:26.868 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-03-25T16:16:26.868 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:16:26.868 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:16:26.868 INFO:teuthology.orchestra.run.smithi086.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:16:26.868 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-25T16:16:26.868 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:16:26.868 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:16:26.868 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:16:26.868 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:16:26.868 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:16:26.868 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:16:26.868 INFO:teuthology.orchestra.run.smithi086.stdout:Install 4 Packages 2024-03-25T16:16:26.868 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:16:26.869 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 76 M 2024-03-25T16:16:26.869 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 123 M 2024-03-25T16:16:26.869 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:16:27.067 INFO:teuthology.orchestra.run.smithi086.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 328 kB/s | 44 kB 00:00 2024-03-25T16:16:27.577 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:16:27.693 INFO:teuthology.orchestra.run.smithi086.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-03-25T16:16:28.176 INFO:teuthology.orchestra.run.smithi086.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 27 MB/s | 33 MB 00:01 2024-03-25T16:16:28.194 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:16:28.281 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:16:29.235 INFO:teuthology.orchestra.run.smithi086.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 13 MB/s | 27 MB 00:02 2024-03-25T16:16:29.235 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:16:29.235 INFO:teuthology.orchestra.run.smithi086.stdout:Total 32 MB/s | 76 MB 00:02 2024-03-25T16:16:29.894 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:16:30.236 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:16:30.236 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:16:31.199 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:16:31.232 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:16:31.233 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:16:31.332 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:16:33.675 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:16:34.262 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:16:34.343 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:16:36.433 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:16:36.466 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:37.129 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:16:37.261 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:16:41.591 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:41.944 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:16:41.986 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:46.696 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:12.486 INFO:teuthology.orchestra.run.smithi064.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:17:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:17:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:17:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:17:13.039 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:17:13.469 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:13.470 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:17:13.470 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:17:13.470 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:17:13.470 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:17:13.470 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:17:13.470 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:17:13.470 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:17:13.470 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:17:13.805 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:17:13.905 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:17:13.906 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:17:13.906 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:17:13.906 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:17:13.906 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:17:13.906 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:17:13.906 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:17:13.907 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:17:13.989 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:17:13.990 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:17:13.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo rpm -qi grub2-tools 2024-03-25T16:17:14.028 INFO:teuthology.orchestra.run.smithi064.stdout:Name : grub2-tools 2024-03-25T16:17:14.028 INFO:teuthology.orchestra.run.smithi064.stdout:Epoch : 1 2024-03-25T16:17:14.028 INFO:teuthology.orchestra.run.smithi064.stdout:Version : 2.06 2024-03-25T16:17:14.028 INFO:teuthology.orchestra.run.smithi064.stdout:Release : 68.el9 2024-03-25T16:17:14.028 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: x86_64 2024-03-25T16:17:14.028 INFO:teuthology.orchestra.run.smithi064.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:17:14.028 INFO:teuthology.orchestra.run.smithi064.stdout:Group : Unspecified 2024-03-25T16:17:14.028 INFO:teuthology.orchestra.run.smithi064.stdout:Size : 8264254 2024-03-25T16:17:14.028 INFO:teuthology.orchestra.run.smithi064.stdout:License : GPLv3+ 2024-03-25T16:17:14.028 INFO:teuthology.orchestra.run.smithi064.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:17:14.028 INFO:teuthology.orchestra.run.smithi064.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:17:14.029 INFO:teuthology.orchestra.run.smithi064.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:17:14.029 INFO:teuthology.orchestra.run.smithi064.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:17:14.029 INFO:teuthology.orchestra.run.smithi064.stdout:Packager : builder@centos.org 2024-03-25T16:17:14.029 INFO:teuthology.orchestra.run.smithi064.stdout:Vendor : CentOS 2024-03-25T16:17:14.029 INFO:teuthology.orchestra.run.smithi064.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:17:14.029 INFO:teuthology.orchestra.run.smithi064.stdout:Summary : Support tools for GRUB. 2024-03-25T16:17:14.029 INFO:teuthology.orchestra.run.smithi064.stdout:Description : 2024-03-25T16:17:14.029 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:17:14.029 INFO:teuthology.orchestra.run.smithi064.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:17:14.029 INFO:teuthology.orchestra.run.smithi064.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:17:14.029 INFO:teuthology.orchestra.run.smithi064.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:17:14.029 INFO:teuthology.orchestra.run.smithi064.stdout:hardware devices. 2024-03-25T16:17:14.029 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:17:14.029 INFO:teuthology.orchestra.run.smithi064.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:17:14.030 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:17:14.030 INFO:teuthology.task.kernel:Updating grub on smithi064 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:17:14.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:17:14.760 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2024-03-25T16:17:15.975 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:15.975 INFO:teuthology.orchestra.run.smithi086.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:17:15.975 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:17:16.018 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:16.093 INFO:teuthology.orchestra.run.smithi064.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:17:16.132 INFO:teuthology.orchestra.run.smithi064.stderr:done 2024-03-25T16:17:16.135 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-03-25T16:17:16.150 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.9ReE0pqLxn 2024-03-25T16:17:16.150 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9ReE0pqLxn 2024-03-25T16:17:16.222 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.9ReE0pqLxn 2024-03-25T16:17:16.367 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.9ReE0pqLxn is 6KB 2024-03-25T16:17:16.388 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:16.388 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:17:16.389 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:17:16.389 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:17:16.418 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.9ReE0pqLxn 2024-03-25T16:17:16.435 DEBUG:teuthology.orchestra.run.smithi064:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:17:16.503 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:17:16.503 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:17:16.503 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:17:16.503 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:17:16.505 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:17:16.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2024-03-25T16:17:16.837 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:16.837 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:17:16.837 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:17:16.837 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:17:16.837 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:17:16.837 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:17:16.837 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:17:16.837 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:17:16.838 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:17:17.151 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:17:17.289 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:17:17.289 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:17:17.290 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:17:17.290 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:17:17.290 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:17:17.290 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:17:17.290 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:17:17.290 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:17:17.370 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:17:17.370 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:17:17.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo rpm -qi grub2-tools 2024-03-25T16:17:17.410 INFO:teuthology.orchestra.run.smithi086.stdout:Name : grub2-tools 2024-03-25T16:17:17.411 INFO:teuthology.orchestra.run.smithi086.stdout:Epoch : 1 2024-03-25T16:17:17.411 INFO:teuthology.orchestra.run.smithi086.stdout:Version : 2.06 2024-03-25T16:17:17.411 INFO:teuthology.orchestra.run.smithi086.stdout:Release : 68.el9 2024-03-25T16:17:17.411 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: x86_64 2024-03-25T16:17:17.411 INFO:teuthology.orchestra.run.smithi086.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:17:17.411 INFO:teuthology.orchestra.run.smithi086.stdout:Group : Unspecified 2024-03-25T16:17:17.411 INFO:teuthology.orchestra.run.smithi086.stdout:Size : 8264254 2024-03-25T16:17:17.411 INFO:teuthology.orchestra.run.smithi086.stdout:License : GPLv3+ 2024-03-25T16:17:17.411 INFO:teuthology.orchestra.run.smithi086.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:17:17.411 INFO:teuthology.orchestra.run.smithi086.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:17:17.411 INFO:teuthology.orchestra.run.smithi086.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:17:17.411 INFO:teuthology.orchestra.run.smithi086.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:17:17.411 INFO:teuthology.orchestra.run.smithi086.stdout:Packager : builder@centos.org 2024-03-25T16:17:17.412 INFO:teuthology.orchestra.run.smithi086.stdout:Vendor : CentOS 2024-03-25T16:17:17.412 INFO:teuthology.orchestra.run.smithi086.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:17:17.412 INFO:teuthology.orchestra.run.smithi086.stdout:Summary : Support tools for GRUB. 2024-03-25T16:17:17.412 INFO:teuthology.orchestra.run.smithi086.stdout:Description : 2024-03-25T16:17:17.412 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:17:17.412 INFO:teuthology.orchestra.run.smithi086.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:17:17.412 INFO:teuthology.orchestra.run.smithi086.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:17:17.412 INFO:teuthology.orchestra.run.smithi086.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:17:17.412 INFO:teuthology.orchestra.run.smithi086.stdout:hardware devices. 2024-03-25T16:17:17.412 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:17:17.412 INFO:teuthology.orchestra.run.smithi086.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:17:17.413 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:17:17.413 INFO:teuthology.task.kernel:Updating grub on smithi086 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:17:17.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:17:18.067 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2024-03-25T16:17:19.453 INFO:teuthology.orchestra.run.smithi086.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:17:19.499 INFO:teuthology.orchestra.run.smithi086.stderr:done 2024-03-25T16:17:19.501 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-03-25T16:17:19.516 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.rjIF4TGZjE 2024-03-25T16:17:19.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rjIF4TGZjE 2024-03-25T16:17:19.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.rjIF4TGZjE 2024-03-25T16:17:19.725 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.rjIF4TGZjE is 6KB 2024-03-25T16:17:19.773 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.rjIF4TGZjE 2024-03-25T16:17:19.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:17:19.859 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:17:19.859 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:17:19.859 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:17:19.859 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:17:19.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:17:20.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2024-03-25T16:17:46.674 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:17:46.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-25T16:17:46.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:17:50.014 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:17:50.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-25T16:17:50.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:18:05.215 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-03-25T16:18:14.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-25T16:18:14.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:18:50.054 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:18:59.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-25T16:18:59.065 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:18:59.595 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-03-25T16:19:00.063 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-25T16:19:00.063 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:19:00.063 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-03-25T16:19:00.078 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:19:00.081 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:19:00.081 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:19:00.082 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:19:01.082 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:19:01.082 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:19:01.140 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2024-03-25T16:19:01.177 DEBUG:teuthology.parallel:result is None 2024-03-25T16:19:14.225 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:19:26.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-25T16:19:26.227 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:19:26.515 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-03-25T16:19:26.892 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-25T16:19:26.892 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:19:26.893 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-03-25T16:19:26.907 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:19:26.908 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:19:26.908 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:19:26.908 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:19:27.908 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:19:27.909 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:19:27.997 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2024-03-25T16:19:28.031 DEBUG:teuthology.parallel:result is None 2024-03-25T16:19:28.031 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:19:28.039 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:19:28.039 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:19:28.041 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:19:28.060 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:19:28.067 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:19:28.074 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:19:28.075 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:19:28.099 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:19:28.166 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:19:28.172 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:19:28.173 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:19:28.175 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:19:28.209 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:19:28.217 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:19:28.222 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:19:28.231 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:19:28.233 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:19:28.279 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:19:28.279 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:19:28.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:19:28.311 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:19:28.320 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:19:28.320 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:19:28.353 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:19:28.369 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:19:28.451 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:19:28.500 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:19:28.581 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:19:28.624 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T16:19:28.624 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:19:28.691 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:19:28.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:19:28.788 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:19:28.872 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:19:28.917 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T16:19:28.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:19:28.986 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-03-25T16:19:28.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-03-25T16:19:29.034 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:19:29.094 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:19:29.523 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:19:29.531 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:19:29.531 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:19:29.564 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:19:29.573 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-03-25T16:19:29.650 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-03-25T16:19:29.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2024-03-25T16:19:29.713 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2024-03-25T16:19:29.715 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:19:29.716 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-03-25T16:19:29.737 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-03-25T16:19:29.737 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2024-03-25T16:19:29.762 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2024-03-25T16:19:29.762 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi086.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:19:29.762 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:19:29.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:19:29.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:19:29.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:19:29.836 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:19:29.836 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:19:29.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:19:29.922 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:19:29.933 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:19:29.951 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:19:29.952 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi064.front.sepia.ceph.com,smithi086.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:23:52.463 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi086.front.sepia.ceph.com')] 2024-03-25T16:23:52.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-25T16:23:52.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:23:52.563 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-03-25T16:23:52.637 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-25T16:23:52.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-25T16:23:52.638 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:23:52.725 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-03-25T16:23:52.801 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-25T16:23:52.802 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:23:52.839 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:23:52.839 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:23:52.840 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:23:52.842 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:23:52.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:23:52.872 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:23:52.874 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:23:52.889 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:23:52.891 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:23:52.919 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-03-25T16:23:52.921 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2024-03-25T16:23:52.933 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-03-25T16:23:52.934 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2024-03-25T16:23:52.949 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:23:52.951 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:23:52.966 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:23:52.967 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:23:53.021 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:23:53.025 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:23:53.025 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-03-25T16:23:53.026 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:23:53.026 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:23:53.026 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:23:53.026 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:23:53.026 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:23:53.030 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:23:53.030 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-03-25T16:23:53.030 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:23:53.030 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:23:53.030 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:23:53.030 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:23:53.031 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:23:53.044 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:23:53.044 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:23:53.044 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:23:53.044 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:23:53.052 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:23:53.052 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:23:53.053 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:23:53.053 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:23:53.055 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:23:53.055 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:23:53.199 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:23:53.200 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:23:53.203 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:23:53.204 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:23:53.544 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:23:53.544 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T16:23:53.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:23:53.574 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:23:53.574 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:23:53.600 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:23:53.600 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T16:23:53.600 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:23:53.629 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:23:53.630 DEBUG:teuthology.orchestra.run.smithi064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:23:53.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:23:53.698 DEBUG:teuthology.orchestra.run.smithi064:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:23:53.724 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:23:53.777 DEBUG:teuthology.orchestra.run.smithi064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:23:53.792 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2024-03-25T16:23:53.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2024-03-25T16:23:53.844 INFO:teuthology.orchestra.run.smithi064.stdout:check_obsoletes = 1 2024-03-25T16:23:53.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum clean all 2024-03-25T16:23:54.040 INFO:teuthology.orchestra.run.smithi086.stdout:74 files removed 2024-03-25T16:23:54.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw 2024-03-25T16:23:54.102 INFO:teuthology.orchestra.run.smithi064.stdout:74 files removed 2024-03-25T16:23:54.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw 2024-03-25T16:23:55.272 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 91 kB/s | 82 kB 00:00 2024-03-25T16:23:55.465 INFO:teuthology.orchestra.run.smithi064.stdout:ceph packages for x86_64 79 kB/s | 82 kB 00:01 2024-03-25T16:23:56.139 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T16:23:56.390 INFO:teuthology.orchestra.run.smithi064.stdout:ceph noarch packages 11 kB/s | 10 kB 00:00 2024-03-25T16:23:56.964 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T16:23:57.224 INFO:teuthology.orchestra.run.smithi064.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T16:23:57.392 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-25T16:23:57.893 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T16:23:58.809 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-25T16:23:59.402 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-25T16:24:00.300 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 6.9 MB/s | 8.0 MB 00:01 2024-03-25T16:24:02.328 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-25T16:24:07.237 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.1 MB 00:01 2024-03-25T16:24:08.738 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-25T16:24:09.317 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 859 kB/s | 8.0 MB 00:09 2024-03-25T16:24:09.922 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-25T16:24:10.962 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 33 MB/s | 19 MB 00:00 2024-03-25T16:24:14.915 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-03-25T16:24:15.352 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 7.8 MB/s | 6.1 MB 00:00 2024-03-25T16:24:16.836 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-03-25T16:24:17.375 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:24:17.398 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:24:17.398 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-25T16:24:17.398 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:24:17.398 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:24:17.398 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:24:17.398 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2024-03-25T16:24:17.398 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:24:17.398 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:24:17.399 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-25T16:24:17.399 INFO:teuthology.orchestra.run.smithi086.stdout: boost-program-options 2024-03-25T16:24:17.399 INFO:teuthology.orchestra.run.smithi086.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:24:17.399 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:24:17.399 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:24:17.399 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:24:17.399 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:24:17.399 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:24:17.399 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:24:17.399 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:24:17.399 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:24:17.399 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:24:17.399 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:24:17.399 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:24:17.400 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:24:17.400 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:24:17.400 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:24:17.400 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:24:17.400 INFO:teuthology.orchestra.run.smithi086.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:24:17.400 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse 2024-03-25T16:24:17.400 INFO:teuthology.orchestra.run.smithi086.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:24:17.400 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:24:17.400 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:24:17.400 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:24:17.400 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:24:17.400 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:24:17.400 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:24:17.400 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:24:17.401 INFO:teuthology.orchestra.run.smithi086.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:24:17.401 INFO:teuthology.orchestra.run.smithi086.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:24:17.401 INFO:teuthology.orchestra.run.smithi086.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:24:17.401 INFO:teuthology.orchestra.run.smithi086.stdout:Installing weak dependencies: 2024-03-25T16:24:17.401 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:24:17.401 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:24:17.401 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:24:17.401 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:24:17.401 INFO:teuthology.orchestra.run.smithi086.stdout:Install 30 Packages 2024-03-25T16:24:17.401 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 2 Packages 2024-03-25T16:24:17.401 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:24:17.402 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 58 M 2024-03-25T16:24:17.402 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:24:18.120 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-25T16:24:22.449 INFO:teuthology.orchestra.run.smithi086.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.1 MB/s | 5.3 MB 00:04 2024-03-25T16:24:22.540 INFO:teuthology.orchestra.run.smithi086.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 275 kB/s | 25 kB 00:00 2024-03-25T16:24:23.091 INFO:teuthology.orchestra.run.smithi086.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 742 kB 00:00 2024-03-25T16:24:23.190 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-03-25T16:24:23.508 INFO:teuthology.orchestra.run.smithi086.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.2 MB/s | 498 kB 00:00 2024-03-25T16:24:25.738 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:24:25.742 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:24:25.743 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-25T16:24:25.743 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:24:25.743 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:24:25.743 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:24:25.743 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrading: 2024-03-25T16:24:25.743 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:24:25.743 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:24:25.743 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-25T16:24:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options 2024-03-25T16:24:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:24:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:24:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:24:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:24:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:24:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:24:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:24:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:24:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:24:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:24:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:24:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:24:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:24:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:24:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:24:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:24:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:24:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse 2024-03-25T16:24:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:24:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:24:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:24:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:24:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:24:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:24:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:24:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:24:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:24:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:24:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:24:25.746 INFO:teuthology.orchestra.run.smithi064.stdout:Installing weak dependencies: 2024-03-25T16:24:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:24:25.747 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:24:25.747 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:24:25.747 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:24:25.747 INFO:teuthology.orchestra.run.smithi064.stdout:Install 30 Packages 2024-03-25T16:24:25.747 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrade 2 Packages 2024-03-25T16:24:25.747 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:24:25.747 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 58 M 2024-03-25T16:24:25.747 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:24:27.416 INFO:teuthology.orchestra.run.smithi086.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 1.3 MB/s | 5.1 MB 00:03 2024-03-25T16:24:27.516 INFO:teuthology.orchestra.run.smithi086.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 452 kB/s | 45 kB 00:00 2024-03-25T16:24:28.033 INFO:teuthology.orchestra.run.smithi086.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 254 kB/s | 131 kB 00:00 2024-03-25T16:24:28.300 INFO:teuthology.orchestra.run.smithi086.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 607 kB/s | 162 kB 00:00 2024-03-25T16:24:28.800 INFO:teuthology.orchestra.run.smithi086.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 641 kB/s | 320 kB 00:00 2024-03-25T16:24:29.241 INFO:teuthology.orchestra.run.smithi086.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 676 kB/s | 298 kB 00:00 2024-03-25T16:24:29.676 INFO:teuthology.orchestra.run.smithi086.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 229 kB/s | 99 kB 00:00 2024-03-25T16:24:29.785 INFO:teuthology.orchestra.run.smithi086.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-03-25T16:24:29.827 INFO:teuthology.orchestra.run.smithi086.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T16:24:29.860 INFO:teuthology.orchestra.run.smithi086.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:24:29.919 INFO:teuthology.orchestra.run.smithi086.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T16:24:29.961 INFO:teuthology.orchestra.run.smithi086.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:24:29.994 INFO:teuthology.orchestra.run.smithi086.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:24:30.086 INFO:teuthology.orchestra.run.smithi086.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 2.3 MB/s | 216 kB 00:00 2024-03-25T16:24:30.136 INFO:teuthology.orchestra.run.smithi086.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 700 kB/s | 35 kB 00:00 2024-03-25T16:24:30.195 INFO:teuthology.orchestra.run.smithi086.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 763 kB/s | 44 kB 00:00 2024-03-25T16:24:30.587 INFO:teuthology.orchestra.run.smithi086.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 789 kB/s | 309 kB 00:00 2024-03-25T16:24:30.879 INFO:teuthology.orchestra.run.smithi086.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-03-25T16:24:30.979 INFO:teuthology.orchestra.run.smithi086.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-03-25T16:24:31.080 INFO:teuthology.orchestra.run.smithi086.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-03-25T16:24:31.163 INFO:teuthology.orchestra.run.smithi086.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 811 kB/s | 67 kB 00:00 2024-03-25T16:24:31.272 INFO:teuthology.orchestra.run.smithi086.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 838 kB 00:00 2024-03-25T16:24:31.338 INFO:teuthology.orchestra.run.smithi086.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-25T16:24:31.455 INFO:teuthology.orchestra.run.smithi086.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-25T16:24:37.064 INFO:teuthology.orchestra.run.smithi086.stdout:(29/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 616 kB/s | 3.4 MB 00:05 2024-03-25T16:24:42.897 INFO:teuthology.orchestra.run.smithi086.stdout:(30/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 394 kB/s | 9.7 MB 00:25 2024-03-25T16:24:43.581 INFO:teuthology.orchestra.run.smithi086.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 492 kB/s | 3.1 MB 00:06 2024-03-25T16:24:44.914 INFO:teuthology.orchestra.run.smithi064.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 290 kB/s | 5.3 MB 00:18 2024-03-25T16:24:45.206 INFO:teuthology.orchestra.run.smithi064.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 86 kB/s | 25 kB 00:00 2024-03-25T16:24:47.289 INFO:teuthology.orchestra.run.smithi064.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 356 kB/s | 742 kB 00:02 2024-03-25T16:24:49.006 INFO:teuthology.orchestra.run.smithi064.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 290 kB/s | 498 kB 00:01 2024-03-25T16:24:50.695 INFO:teuthology.orchestra.run.smithi064.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 404 kB/s | 9.7 MB 00:24 2024-03-25T16:24:50.814 INFO:teuthology.orchestra.run.smithi064.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 380 kB/s | 45 kB 00:00 2024-03-25T16:24:51.115 INFO:teuthology.orchestra.run.smithi064.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 438 kB/s | 131 kB 00:00 2024-03-25T16:24:51.507 INFO:teuthology.orchestra.run.smithi064.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 413 kB/s | 162 kB 00:00 2024-03-25T16:24:52.282 INFO:teuthology.orchestra.run.smithi064.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 413 kB/s | 320 kB 00:00 2024-03-25T16:24:53.141 INFO:teuthology.orchestra.run.smithi064.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 347 kB/s | 298 kB 00:00 2024-03-25T16:24:53.449 INFO:teuthology.orchestra.run.smithi064.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 322 kB/s | 99 kB 00:00 2024-03-25T16:24:53.549 INFO:teuthology.orchestra.run.smithi064.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2024-03-25T16:24:53.591 INFO:teuthology.orchestra.run.smithi064.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T16:24:53.625 INFO:teuthology.orchestra.run.smithi064.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:24:53.675 INFO:teuthology.orchestra.run.smithi064.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-25T16:24:53.717 INFO:teuthology.orchestra.run.smithi064.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:24:53.751 INFO:teuthology.orchestra.run.smithi064.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:24:53.968 INFO:teuthology.orchestra.run.smithi064.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.0 MB/s | 216 kB 00:00 2024-03-25T16:24:54.027 INFO:teuthology.orchestra.run.smithi064.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 596 kB/s | 35 kB 00:00 2024-03-25T16:24:54.085 INFO:teuthology.orchestra.run.smithi064.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 761 kB/s | 44 kB 00:00 2024-03-25T16:24:54.644 INFO:teuthology.orchestra.run.smithi064.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 553 kB/s | 309 kB 00:00 2024-03-25T16:24:55.998 INFO:teuthology.orchestra.run.smithi064.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 4.4 MB 00:01 2024-03-25T16:24:56.044 INFO:teuthology.orchestra.run.smithi064.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 566 kB/s | 26 kB 00:00 2024-03-25T16:24:56.094 INFO:teuthology.orchestra.run.smithi064.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 974 kB/s | 49 kB 00:00 2024-03-25T16:24:56.153 INFO:teuthology.orchestra.run.smithi064.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-25T16:24:56.295 INFO:teuthology.orchestra.run.smithi064.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-03-25T16:24:56.345 INFO:teuthology.orchestra.run.smithi064.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.8 MB/s | 192 kB 00:00 2024-03-25T16:24:56.521 INFO:teuthology.orchestra.run.smithi064.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-25T16:25:04.796 INFO:teuthology.orchestra.run.smithi086.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 431 kB/s | 20 MB 00:47 2024-03-25T16:25:04.800 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:04.801 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.2 MB/s | 58 MB 00:47 2024-03-25T16:25:04.832 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:25:04.871 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:25:04.871 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:25:05.229 INFO:teuthology.orchestra.run.smithi064.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 322 kB/s | 5.1 MB 00:16 2024-03-25T16:25:05.352 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:25:05.352 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:25:06.807 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:25:06.916 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:25:06.959 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:25:06.991 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:25:07.012 INFO:teuthology.orchestra.run.smithi064.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 329 kB/s | 3.4 MB 00:10 2024-03-25T16:25:07.015 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:25:07.032 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:25:07.199 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:25:07.254 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:25:07.584 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:25:07.636 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:25:07.719 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:25:07.806 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:25:07.829 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:25:07.875 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:25:07.903 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:25:07.939 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:25:08.006 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:25:08.031 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:25:08.094 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:25:08.121 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:25:08.229 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:25:08.272 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:25:08.321 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:25:08.354 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:25:08.383 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:25:08.415 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:25:08.741 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:25:08.771 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:25:08.843 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:25:08.875 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:25:08.917 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:25:09.365 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:25:09.390 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:25:09.460 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:25:09.529 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:25:09.567 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:25:11.747 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:25:11.796 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:25:12.312 INFO:teuthology.orchestra.run.smithi064.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 452 kB/s | 3.1 MB 00:07 2024-03-25T16:25:12.340 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:25:12.383 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:25:12.488 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:25:12.489 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:25:12.489 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:25:12.489 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:25:12.517 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:25:21.378 INFO:teuthology.orchestra.run.smithi064.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 368 kB/s | 20 MB 00:55 2024-03-25T16:25:21.381 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:21.382 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.0 MB/s | 58 MB 00:55 2024-03-25T16:25:21.412 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:25:21.453 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:25:21.453 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:25:21.776 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:25:21.777 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2024-03-25T16:25:21.777 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2024-03-25T16:25:21.777 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2024-03-25T16:25:21.777 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2024-03-25T16:25:21.777 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2024-03-25T16:25:21.777 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /root 2024-03-25T16:25:21.777 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2024-03-25T16:25:21.777 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:25:21.952 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:25:21.952 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:25:22.687 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:25:22.711 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:25:22.711 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:22.711 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:25:22.711 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:25:22.711 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:25:22.711 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:25:22.713 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:25:22.798 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:25:22.799 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:25:23.611 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:25:23.687 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:25:23.715 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:25:23.740 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:25:23.763 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:25:23.779 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:25:23.923 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:25:23.923 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:25:23.923 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:25:23.923 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:25:23.923 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:25:23.923 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:25:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:25:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:25:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:25:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:25:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:25:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:25:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:25:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:25:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:25:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:25:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:25:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:25:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:25:23.924 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:25:23.925 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:25:23.925 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:25:23.925 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:25:23.925 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:25:23.925 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:25:23.925 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:25:23.925 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:25:23.925 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:25:23.926 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:25:23.926 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:25:23.926 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:25:23.926 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:25:23.926 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:25:23.926 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:25:23.981 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:25:24.036 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:25:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:25:24.478 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:25:24.571 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:25:24.666 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:25:24.685 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:25:24.748 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:25:24.777 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:25:24.813 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:25:24.882 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:25:24.904 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:25:24.922 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:25:24.922 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:25:24.922 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2024-03-25T16:25:24.922 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:24.922 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:24.922 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:25:24.922 INFO:teuthology.orchestra.run.smithi086.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:25:24.922 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:24.922 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:24.922 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:24.923 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:24.923 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:25:24.923 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:25:24.923 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:25:24.923 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:24.923 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:25:24.923 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:25:24.923 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:25:24.923 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:24.923 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:25:24.923 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:24.923 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:25:24.923 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:25:24.923 INFO:teuthology.orchestra.run.smithi086.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:25:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:25:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:25:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:25:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:25:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:25:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:25:24.924 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:25:24.924 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:25:24.950 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:25:24.977 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:25:25.090 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:25:25.120 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:25:25.162 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:25:25.193 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:25:25.223 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:25:25.255 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:25:25.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-test 2024-03-25T16:25:25.590 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:25:25.626 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:25:25.717 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:25:25.748 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:25:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:25:25.868 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:11 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:25:26.015 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:25:26.015 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:25:26.016 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-25T16:25:26.016 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:25:26.016 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:25:26.016 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:25:26.016 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-25T16:25:26.016 INFO:teuthology.orchestra.run.smithi086.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:25:26.016 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:25:26.016 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:25:26.016 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:25:26.016 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:25:26.016 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:25:26.016 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:25:26.016 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:25:26.016 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-03-25T16:25:26.017 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:25:26.017 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 48 M 2024-03-25T16:25:26.017 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 204 M 2024-03-25T16:25:26.017 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:25:26.265 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:25:26.287 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:25:26.358 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:25:26.429 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:25:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:25:26.470 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T16:25:26.520 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T16:25:26.554 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T16:25:26.771 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 864 kB/s | 187 kB 00:00 2024-03-25T16:25:27.704 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 117 kB/s | 163 kB 00:01 2024-03-25T16:25:28.721 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:25:28.761 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:25:29.332 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:25:29.364 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:25:29.466 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:25:29.466 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:25:29.466 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:25:29.467 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:25:29.507 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:25:39.397 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:25:39.397 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /sys 2024-03-25T16:25:39.397 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /proc 2024-03-25T16:25:39.398 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /mnt 2024-03-25T16:25:39.398 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /var/tmp 2024-03-25T16:25:39.398 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /home 2024-03-25T16:25:39.398 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /root 2024-03-25T16:25:39.398 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /tmp 2024-03-25T16:25:39.398 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:25:40.326 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:25:40.350 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:25:40.350 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:40.350 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:25:40.350 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:25:40.350 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:25:40.350 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:25:40.351 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:25:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:25:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:25:41.601 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:25:41.601 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:25:41.601 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:25:41.601 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:25:41.601 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:25:41.601 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:25:41.601 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:25:41.601 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:25:41.601 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:25:41.602 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:25:41.602 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:25:41.602 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:25:41.602 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:25:41.602 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:25:41.602 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:25:41.602 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:25:41.602 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:25:41.602 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:25:41.602 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:25:41.602 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:25:41.602 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:25:41.603 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:25:41.603 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:25:41.603 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:25:41.603 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:25:41.603 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:25:41.603 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:25:41.603 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:25:41.603 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:25:41.603 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:25:41.603 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:25:41.603 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:25:41.603 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:25:41.603 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:25:42.863 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:25:42.863 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:25:42.863 INFO:teuthology.orchestra.run.smithi064.stdout:Upgraded: 2024-03-25T16:25:42.863 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:42.863 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:42.863 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:25:42.863 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:25:42.863 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:42.863 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:42.863 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:42.863 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:42.864 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:25:42.864 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:25:42.864 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:25:42.864 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:42.864 INFO:teuthology.orchestra.run.smithi064.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:25:42.864 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:25:42.864 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:25:42.864 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:42.864 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:25:42.864 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:42.864 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:25:42.864 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:25:42.864 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:25:42.864 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:42.865 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:42.865 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:42.865 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:25:42.865 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:42.865 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:42.865 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:42.865 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:25:42.865 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:25:42.865 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:25:42.865 INFO:teuthology.orchestra.run.smithi064.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:25:42.865 INFO:teuthology.orchestra.run.smithi064.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:25:42.865 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:25:42.865 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:25:43.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-test 2024-03-25T16:25:43.716 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:20 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:25:43.874 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:25:43.875 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:25:43.875 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-25T16:25:43.875 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:25:43.875 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:25:43.875 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:25:43.875 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-25T16:25:43.875 INFO:teuthology.orchestra.run.smithi064.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:25:43.876 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:25:43.876 INFO:teuthology.orchestra.run.smithi064.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:25:43.876 INFO:teuthology.orchestra.run.smithi064.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:25:43.876 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:25:43.876 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:25:43.876 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:25:43.876 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:25:43.876 INFO:teuthology.orchestra.run.smithi064.stdout:Install 6 Packages 2024-03-25T16:25:43.876 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:25:43.877 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 48 M 2024-03-25T16:25:43.877 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 204 M 2024-03-25T16:25:43.877 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:25:44.074 INFO:teuthology.orchestra.run.smithi064.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.6 MB/s | 217 kB 00:00 2024-03-25T16:25:44.149 INFO:teuthology.orchestra.run.smithi064.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T16:25:44.183 INFO:teuthology.orchestra.run.smithi064.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T16:25:44.717 INFO:teuthology.orchestra.run.smithi064.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 211 kB/s | 163 kB 00:00 2024-03-25T16:25:45.017 INFO:teuthology.orchestra.run.smithi064.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 225 kB/s | 187 kB 00:00 2024-03-25T16:26:30.582 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 753 kB/s | 47 MB 01:04 2024-03-25T16:26:30.585 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:30.585 INFO:teuthology.orchestra.run.smithi086.stdout:Total 764 kB/s | 48 MB 01:04 2024-03-25T16:26:30.597 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:26:30.617 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:26:30.617 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:26:31.000 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:26:31.000 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:26:31.597 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:26:31.673 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:26:31.713 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:26:31.753 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:26:31.797 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:26:31.818 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:26:36.306 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:26:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:26:38.651 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:26:38.651 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:26:38.651 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:26:38.652 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:26:38.652 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:26:38.652 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:26:39.009 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:26:39.009 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:26:39.010 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:26:39.010 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:39.010 INFO:teuthology.orchestra.run.smithi086.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:26:39.010 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:39.010 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:26:39.010 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:26:39.010 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:26:39.010 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:26:39.010 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:26:39.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph 2024-03-25T16:26:39.892 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:25 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:26:40.044 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:26:40.047 INFO:teuthology.orchestra.run.smithi086.stdout:========================================================================================= 2024-03-25T16:26:40.047 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-25T16:26:40.047 INFO:teuthology.orchestra.run.smithi086.stdout:========================================================================================= 2024-03-25T16:26:40.047 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:26:40.048 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:26:40.048 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-25T16:26:40.048 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:26:40.048 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:26:40.048 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:26:40.048 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:26:40.048 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:26:40.048 INFO:teuthology.orchestra.run.smithi086.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:26:40.048 INFO:teuthology.orchestra.run.smithi086.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:26:40.048 INFO:teuthology.orchestra.run.smithi086.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:26:40.048 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:26:40.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:26:40.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:26:40.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:26:40.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:26:40.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:26:40.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:26:40.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:26:40.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:26:40.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:26:40.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:26:40.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:26:40.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:26:40.049 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:26:40.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:26:40.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:26:40.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:26:40.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:26:40.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:26:40.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:26:40.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:26:40.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:26:40.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:26:40.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:26:40.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:26:40.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:26:40.050 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:26:40.051 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:26:40.051 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:26:40.051 INFO:teuthology.orchestra.run.smithi086.stdout:========================================================================================= 2024-03-25T16:26:40.051 INFO:teuthology.orchestra.run.smithi086.stdout:Install 36 Packages 2024-03-25T16:26:40.051 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:26:40.051 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 30 M 2024-03-25T16:26:40.051 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 107 M 2024-03-25T16:26:40.051 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:26:40.868 INFO:teuthology.orchestra.run.smithi086.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-25T16:26:43.318 INFO:teuthology.orchestra.run.smithi086.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 772 kB/s | 2.1 MB 00:02 2024-03-25T16:26:43.693 INFO:teuthology.orchestra.run.smithi086.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 477 kB/s | 1.5 MB 00:03 2024-03-25T16:26:44.244 INFO:teuthology.orchestra.run.smithi086.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 450 kB/s | 247 kB 00:00 2024-03-25T16:26:44.552 INFO:teuthology.orchestra.run.smithi086.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 606 kB/s | 187 kB 00:00 2024-03-25T16:26:44.652 INFO:teuthology.orchestra.run.smithi086.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2024-03-25T16:26:44.719 INFO:teuthology.orchestra.run.smithi086.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 473 kB/s | 32 kB 00:00 2024-03-25T16:26:44.778 INFO:teuthology.orchestra.run.smithi086.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 810 kB/s | 47 kB 00:00 2024-03-25T16:26:44.837 INFO:teuthology.orchestra.run.smithi086.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 663 kB/s | 39 kB 00:00 2024-03-25T16:26:44.945 INFO:teuthology.orchestra.run.smithi086.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.2 MB/s | 241 kB 00:00 2024-03-25T16:26:45.062 INFO:teuthology.orchestra.run.smithi086.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 9.8 MB/s | 1.1 MB 00:00 2024-03-25T16:26:45.104 INFO:teuthology.orchestra.run.smithi086.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T16:26:45.146 INFO:teuthology.orchestra.run.smithi086.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T16:26:45.188 INFO:teuthology.orchestra.run.smithi086.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:26:45.338 INFO:teuthology.orchestra.run.smithi086.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 833 kB/s | 125 kB 00:00 2024-03-25T16:26:45.389 INFO:teuthology.orchestra.run.smithi086.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 447 kB/s | 22 kB 00:00 2024-03-25T16:26:45.833 INFO:teuthology.orchestra.run.smithi086.stdout:(17/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 966 kB/s | 4.7 MB 00:04 2024-03-25T16:26:45.858 INFO:teuthology.orchestra.run.smithi086.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 322 kB/s | 151 kB 00:00 2024-03-25T16:26:46.050 INFO:teuthology.orchestra.run.smithi086.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 896 kB/s | 172 kB 00:00 2024-03-25T16:26:46.150 INFO:teuthology.orchestra.run.smithi086.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 137 kB/s | 43 kB 00:00 2024-03-25T16:26:46.242 INFO:teuthology.orchestra.run.smithi086.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-03-25T16:26:46.259 INFO:teuthology.orchestra.run.smithi086.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 99 kB/s | 11 kB 00:00 2024-03-25T16:26:46.334 INFO:teuthology.orchestra.run.smithi086.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-03-25T16:26:46.368 INFO:teuthology.orchestra.run.smithi086.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2024-03-25T16:26:46.427 INFO:teuthology.orchestra.run.smithi086.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 212 kB/s | 19 kB 00:00 2024-03-25T16:26:46.477 INFO:teuthology.orchestra.run.smithi086.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-03-25T16:26:46.519 INFO:teuthology.orchestra.run.smithi086.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 504 kB/s | 46 kB 00:00 2024-03-25T16:26:46.644 INFO:teuthology.orchestra.run.smithi086.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-25T16:26:46.679 INFO:teuthology.orchestra.run.smithi086.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 410 kB/s | 79 kB 00:00 2024-03-25T16:26:46.737 INFO:teuthology.orchestra.run.smithi086.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 178 kB/s | 16 kB 00:00 2024-03-25T16:26:46.820 INFO:teuthology.orchestra.run.smithi086.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 599 kB/s | 90 kB 00:00 2024-03-25T16:26:46.837 INFO:teuthology.orchestra.run.smithi086.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 357 kB/s | 36 kB 00:00 2024-03-25T16:26:46.987 INFO:teuthology.orchestra.run.smithi086.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.8 MB/s | 427 kB 00:00 2024-03-25T16:26:47.029 INFO:teuthology.orchestra.run.smithi086.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.1 MB/s | 230 kB 00:00 2024-03-25T16:26:47.079 INFO:teuthology.orchestra.run.smithi086.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-25T16:27:10.085 INFO:teuthology.orchestra.run.smithi086.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 642 kB/s | 17 MB 00:26 2024-03-25T16:27:10.089 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:10.089 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.0 MB/s | 30 MB 00:30 2024-03-25T16:27:10.112 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:27:10.155 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:27:10.155 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:27:10.554 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:27:10.554 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:27:11.881 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:27:11.946 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:27:11.980 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:27:12.006 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:27:12.077 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:27:12.130 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:27:12.163 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:27:12.255 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:27:12.358 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:27:12.399 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:27:12.470 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:27:12.524 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:27:12.563 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:27:12.619 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:27:12.674 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:27:12.866 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:27:12.925 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:27:12.955 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:27:12.990 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:27:13.020 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:27:13.095 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:27:13.138 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:27:13.172 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:27:13.229 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:27:13.264 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:27:13.298 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:27:13.387 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:27:13.420 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:27:13.492 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:27:13.627 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:27:13.754 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:27:13.921 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:27:13.950 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:27:13.976 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:27:13.976 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:27:13.976 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:27:13.976 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:27:13.976 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:27:13.976 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:15.444 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:27:15.471 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:27:15.471 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:27:15.471 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:27:15.471 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:27:15.471 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:27:15.471 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:15.911 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:27:15.934 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:27:15.934 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:27:15.934 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:27:15.934 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:27:15.934 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:27:15.935 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:16.137 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:27:16.159 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:27:16.160 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:27:16.160 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:27:16.160 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:27:16.160 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:27:16.160 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:16.226 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:27:17.767 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:27:17.767 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:27:17.767 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:27:17.767 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:27:17.767 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:27:17.767 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:27:17.767 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:27:17.767 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:27:17.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:27:17.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:27:17.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:27:17.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:27:17.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:27:17.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:27:17.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:27:17.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:27:17.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:27:17.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:27:17.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:27:17.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:27:17.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:27:17.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:27:17.768 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:27:17.769 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:27:17.769 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:27:17.769 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:27:17.770 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:27:17.770 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:27:17.770 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:27:17.771 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:27:17.771 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:27:17.771 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:27:17.771 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:27:17.771 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:27:17.771 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:27:17.771 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:27:18.666 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:27:18.666 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:18.666 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:27:18.666 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:18.667 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:18.667 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:18.667 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:18.667 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:18.667 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:18.668 INFO:teuthology.orchestra.run.smithi086.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:27:18.668 INFO:teuthology.orchestra.run.smithi086.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:27:18.668 INFO:teuthology.orchestra.run.smithi086.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:27:18.668 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:27:18.668 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:27:18.668 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:27:18.669 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:27:18.669 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:27:18.669 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:27:18.669 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:27:18.669 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:27:18.669 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:27:18.670 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:27:18.670 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:27:18.670 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:27:18.670 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:27:18.670 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:27:18.670 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:27:18.670 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:27:18.670 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:27:18.670 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:27:18.670 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:27:18.670 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:27:18.671 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:27:18.671 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:27:18.671 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:27:18.671 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:27:18.671 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:27:18.671 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:27:18.671 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:27:18.671 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:18.671 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:27:19.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base 2024-03-25T16:27:19.611 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:05 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:27:19.708 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:19.759 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:27:19.760 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-25T16:27:19.760 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:27:19.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install cephadm 2024-03-25T16:27:20.374 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:06 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:27:20.525 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:27:20.526 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:27:20.526 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-25T16:27:20.526 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:27:20.526 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:27:20.526 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:27:20.526 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:20.526 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:27:20.527 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:27:20.527 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T16:27:20.527 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:20.527 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 747 k 2024-03-25T16:27:20.527 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 754 k 2024-03-25T16:27:20.527 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:27:24.890 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 171 kB/s | 747 kB 00:04 2024-03-25T16:27:24.890 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:24.891 INFO:teuthology.orchestra.run.smithi086.stdout:Total 171 kB/s | 747 kB 00:04 2024-03-25T16:27:24.891 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:27:24.897 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:27:24.897 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:27:24.903 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:27:24.904 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:27:24.982 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:27:25.435 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:27:25.536 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:27:25.909 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:27:26.201 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:27:26.201 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:26.201 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:27:26.201 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:26.201 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:26.201 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:27:26.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:27:26.963 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:12 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:27:27.112 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:27:27.112 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:27:27.112 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T16:27:27.112 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:27:27.113 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:27:27.113 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:27:27.113 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:27.113 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:27:27.113 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:27:27.113 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T16:27:27.113 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:27.113 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 145 k 2024-03-25T16:27:27.113 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 436 k 2024-03-25T16:27:27.113 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:27:28.003 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 163 kB/s | 145 kB 00:00 2024-03-25T16:27:28.004 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:28.004 INFO:teuthology.orchestra.run.smithi086.stdout:Total 163 kB/s | 145 kB 00:00 2024-03-25T16:27:28.004 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:27:28.012 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:27:28.013 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:27:28.066 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:27:28.066 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:27:28.211 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:27:28.270 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:27:28.295 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:27:28.295 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:27:28.295 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:27:28.295 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:29.235 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:27:29.235 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:29.235 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:27:29.235 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:29.236 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:29.236 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:27:29.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr 2024-03-25T16:27:30.081 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:16 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:27:30.179 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:30.234 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:27:30.235 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-25T16:27:30.235 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:27:30.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:27:30.860 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:16 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:27:31.008 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:27:31.009 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2024-03-25T16:27:31.009 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-25T16:27:31.009 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2024-03-25T16:27:31.009 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:27:31.009 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:27:31.009 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-25T16:27:31.009 INFO:teuthology.orchestra.run.smithi086.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:27:31.009 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:27:31.009 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:27:31.009 INFO:teuthology.orchestra.run.smithi086.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:27:31.010 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:27:31.010 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:27:31.010 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:27:31.010 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:27:31.010 INFO:teuthology.orchestra.run.smithi086.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:27:31.010 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:27:31.010 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:27:31.010 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:31.010 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:27:31.010 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2024-03-25T16:27:31.010 INFO:teuthology.orchestra.run.smithi086.stdout:Install 12 Packages 2024-03-25T16:27:31.010 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:31.011 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 6.9 M 2024-03-25T16:27:31.011 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 26 M 2024-03-25T16:27:31.011 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:27:31.788 INFO:teuthology.orchestra.run.smithi086.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 70 kB/s | 31 kB 00:00 2024-03-25T16:27:31.822 INFO:teuthology.orchestra.run.smithi086.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 32 kB/s | 15 kB 00:00 2024-03-25T16:27:31.973 INFO:teuthology.orchestra.run.smithi086.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.8 MB/s | 238 kB 00:00 2024-03-25T16:27:32.006 INFO:teuthology.orchestra.run.smithi086.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-25T16:27:32.157 INFO:teuthology.orchestra.run.smithi086.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.6 MB/s | 863 kB 00:00 2024-03-25T16:27:32.190 INFO:teuthology.orchestra.run.smithi086.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.9 MB/s | 548 kB 00:00 2024-03-25T16:27:32.240 INFO:teuthology.orchestra.run.smithi086.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 233 kB/s | 19 kB 00:00 2024-03-25T16:27:32.332 INFO:teuthology.orchestra.run.smithi086.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.5 MB/s | 144 kB 00:00 2024-03-25T16:27:32.366 INFO:teuthology.orchestra.run.smithi086.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 926 kB/s | 31 kB 00:00 2024-03-25T16:27:32.425 INFO:teuthology.orchestra.run.smithi086.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-03-25T16:27:32.542 INFO:teuthology.orchestra.run.smithi086.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.8 MB/s | 2.0 MB 00:00 2024-03-25T16:27:34.375 INFO:teuthology.orchestra.run.smithi086.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 601 kB/s | 1.8 MB 00:03 2024-03-25T16:27:34.379 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:34.379 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.0 MB/s | 6.9 MB 00:03 2024-03-25T16:27:34.399 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:27:34.417 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:27:34.417 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:27:34.608 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:27:34.609 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:27:35.126 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:27:35.192 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:27:35.227 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:27:35.274 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:27:35.304 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:27:35.408 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:27:35.536 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:27:35.622 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:27:35.666 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:27:35.695 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:27:35.726 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:27:35.971 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:27:36.002 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:27:36.794 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:27:36.794 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:27:36.794 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:27:36.794 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:27:36.795 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:27:36.795 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:27:36.795 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:27:36.795 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:27:36.795 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:27:36.795 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:27:36.795 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:27:36.795 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:27:37.136 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:27:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:37.137 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:27:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:27:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:27:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:27:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:27:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:27:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:27:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:27:37.138 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:27:37.138 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:27:37.138 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:37.138 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:27:37.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:27:37.964 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:23 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:27:38.114 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:27:38.116 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-03-25T16:27:38.116 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-25T16:27:38.116 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-03-25T16:27:38.116 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:27:38.116 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:27:38.116 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-25T16:27:38.116 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:27:38.116 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:27:38.116 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:27:38.116 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:27:38.116 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:27:38.116 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:27:38.117 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:27:38.117 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:27:38.117 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:27:38.117 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:27:38.117 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:27:38.117 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:38.117 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:27:38.117 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-03-25T16:27:38.117 INFO:teuthology.orchestra.run.smithi086.stdout:Install 12 Packages 2024-03-25T16:27:38.117 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:38.118 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 37 M 2024-03-25T16:27:38.118 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 209 M 2024-03-25T16:27:38.118 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:27:38.805 INFO:teuthology.orchestra.run.smithi086.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 228 kB/s | 32 kB 00:00 2024-03-25T16:27:38.830 INFO:teuthology.orchestra.run.smithi086.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 706 kB/s | 17 kB 00:00 2024-03-25T16:27:38.856 INFO:teuthology.orchestra.run.smithi086.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.4 MB/s | 34 kB 00:00 2024-03-25T16:27:38.906 INFO:teuthology.orchestra.run.smithi086.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-25T16:27:39.090 INFO:teuthology.orchestra.run.smithi086.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-25T16:27:39.157 INFO:teuthology.orchestra.run.smithi086.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-25T16:27:39.182 INFO:teuthology.orchestra.run.smithi086.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-25T16:27:39.482 INFO:teuthology.orchestra.run.smithi086.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-03-25T16:27:39.641 INFO:teuthology.orchestra.run.smithi086.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-25T16:27:39.666 INFO:teuthology.orchestra.run.smithi086.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-25T16:27:39.833 INFO:teuthology.orchestra.run.smithi086.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-03-25T16:27:39.979 INFO:teuthology.orchestra.run.smithi064.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 417 kB/s | 47 MB 01:56 2024-03-25T16:27:39.982 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:39.982 INFO:teuthology.orchestra.run.smithi064.stdout:Total 425 kB/s | 48 MB 01:56 2024-03-25T16:27:39.994 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:27:40.014 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:27:40.014 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:27:40.411 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:27:40.412 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:27:41.018 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:27:41.097 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:27:41.136 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:27:41.167 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:27:41.209 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:27:41.232 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:27:45.981 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:27:46.059 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:27:48.543 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:27:48.595 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:27:48.595 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:27:48.595 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:27:48.596 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:27:48.596 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:27:48.945 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:27:48.945 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:27:48.945 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:27:48.946 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:48.946 INFO:teuthology.orchestra.run.smithi064.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:27:48.946 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:48.946 INFO:teuthology.orchestra.run.smithi064.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:27:48.946 INFO:teuthology.orchestra.run.smithi064.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:27:48.946 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:27:48.946 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:27:48.946 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:27:49.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph 2024-03-25T16:27:49.833 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:26 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:27:49.993 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:27:49.996 INFO:teuthology.orchestra.run.smithi064.stdout:========================================================================================= 2024-03-25T16:27:49.996 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-25T16:27:49.996 INFO:teuthology.orchestra.run.smithi064.stdout:========================================================================================= 2024-03-25T16:27:49.996 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:27:49.996 INFO:teuthology.orchestra.run.smithi064.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:27:49.996 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-25T16:27:49.996 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:27:49.996 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:27:49.997 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:27:49.997 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:27:49.997 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:27:49.997 INFO:teuthology.orchestra.run.smithi064.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:27:49.997 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:27:49.997 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:27:49.997 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:27:49.997 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:27:49.997 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:27:49.997 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:27:49.997 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:27:49.998 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:27:49.998 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:27:49.998 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:27:49.998 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:27:49.998 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:27:49.998 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:27:49.998 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:27:49.998 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:27:49.998 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:27:49.998 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:27:49.999 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:27:49.999 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:27:49.999 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:27:49.999 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:27:49.999 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:27:49.999 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:27:49.999 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:27:49.999 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:27:49.999 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:27:49.999 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:27:49.999 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:27:49.999 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:27:50.000 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:27:50.000 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:27:50.000 INFO:teuthology.orchestra.run.smithi064.stdout:========================================================================================= 2024-03-25T16:27:50.000 INFO:teuthology.orchestra.run.smithi064.stdout:Install 36 Packages 2024-03-25T16:27:50.000 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:27:50.000 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 30 M 2024-03-25T16:27:50.001 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 107 M 2024-03-25T16:27:50.001 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:27:50.257 INFO:teuthology.orchestra.run.smithi086.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 653 kB/s | 7.4 MB 00:11 2024-03-25T16:27:50.259 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:50.260 INFO:teuthology.orchestra.run.smithi086.stdout:Total 3.1 MB/s | 37 MB 00:12 2024-03-25T16:27:50.303 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:27:50.328 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:27:50.328 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:27:50.697 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:27:50.698 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:27:50.907 INFO:teuthology.orchestra.run.smithi064.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-25T16:27:51.399 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:27:51.480 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:27:51.511 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:27:51.603 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:27:51.639 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:27:51.816 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:27:51.846 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:27:51.932 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:27:52.569 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:27:52.682 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:27:53.905 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:27:54.698 INFO:teuthology.orchestra.run.smithi064.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 366 kB/s | 1.5 MB 00:04 2024-03-25T16:27:54.927 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:27:54.961 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:27:55.757 INFO:teuthology.orchestra.run.smithi064.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 416 kB/s | 2.1 MB 00:05 2024-03-25T16:27:56.482 INFO:teuthology.orchestra.run.smithi064.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 341 kB/s | 247 kB 00:00 2024-03-25T16:27:56.590 INFO:teuthology.orchestra.run.smithi064.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-25T16:27:56.632 INFO:teuthology.orchestra.run.smithi064.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:27:56.666 INFO:teuthology.orchestra.run.smithi064.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 946 kB/s | 32 kB 00:00 2024-03-25T16:27:56.700 INFO:teuthology.orchestra.run.smithi064.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:27:56.733 INFO:teuthology.orchestra.run.smithi064.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:27:56.775 INFO:teuthology.orchestra.run.smithi064.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T16:27:56.842 INFO:teuthology.orchestra.run.smithi064.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T16:27:56.876 INFO:teuthology.orchestra.run.smithi064.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:27:56.912 INFO:teuthology.orchestra.run.smithi064.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.5 MB/s | 124 kB 00:00 2024-03-25T16:27:56.945 INFO:teuthology.orchestra.run.smithi064.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T16:27:57.403 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:27:57.403 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:27:57.403 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:27:57.403 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:27:57.404 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:27:57.404 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:27:57.404 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:27:57.404 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:27:57.404 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:27:57.404 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:27:57.404 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:27:57.404 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:27:57.721 INFO:teuthology.orchestra.run.smithi064.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 161 kB/s | 125 kB 00:00 2024-03-25T16:27:57.812 INFO:teuthology.orchestra.run.smithi064.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 245 kB/s | 22 kB 00:00 2024-03-25T16:27:57.851 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:27:57.851 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:57.851 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:27:57.851 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:57.851 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:27:57.852 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:27:57.852 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:27:57.852 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:27:57.852 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:27:57.852 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:27:57.852 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:27:57.852 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:27:57.852 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:27:57.852 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:27:57.852 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:27:57.852 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:57.852 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:27:58.104 INFO:teuthology.orchestra.run.smithi064.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 518 kB/s | 151 kB 00:00 2024-03-25T16:27:58.188 INFO:teuthology.orchestra.run.smithi064.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 521 kB/s | 43 kB 00:00 2024-03-25T16:27:58.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:27:58.330 INFO:teuthology.orchestra.run.smithi064.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-03-25T16:27:58.539 INFO:teuthology.orchestra.run.smithi064.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-03-25T16:27:58.622 INFO:teuthology.orchestra.run.smithi064.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-03-25T16:27:58.697 INFO:teuthology.orchestra.run.smithi064.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-03-25T16:27:58.773 INFO:teuthology.orchestra.run.smithi064.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 308 kB/s | 23 kB 00:00 2024-03-25T16:27:58.816 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:44 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:27:58.840 INFO:teuthology.orchestra.run.smithi064.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 291 kB/s | 19 kB 00:00 2024-03-25T16:27:58.898 INFO:teuthology.orchestra.run.smithi064.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 338 kB/s | 20 kB 00:00 2024-03-25T16:27:58.965 INFO:teuthology.orchestra.run.smithi064.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 692 kB/s | 46 kB 00:00 2024-03-25T16:27:58.966 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:27:58.967 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-25T16:27:58.967 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-25T16:27:58.967 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-25T16:27:58.967 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:27:58.967 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:27:58.968 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-25T16:27:58.968 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:27:58.968 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:27:58.968 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:27:58.968 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:27:58.968 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:27:58.968 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:27:58.968 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:27:58.968 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:27:58.968 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:27:58.968 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:27:58.968 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:27:58.968 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:27:58.969 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:58.969 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:27:58.969 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-25T16:27:58.969 INFO:teuthology.orchestra.run.smithi086.stdout:Install 13 Packages 2024-03-25T16:27:58.969 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:27:58.969 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 2.0 M 2024-03-25T16:27:58.969 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 26 M 2024-03-25T16:27:58.970 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:27:59.024 INFO:teuthology.orchestra.run.smithi064.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-25T16:27:59.124 INFO:teuthology.orchestra.run.smithi064.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-25T16:27:59.175 INFO:teuthology.orchestra.run.smithi064.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 329 kB/s | 16 kB 00:00 2024-03-25T16:27:59.233 INFO:teuthology.orchestra.run.smithi064.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-25T16:27:59.283 INFO:teuthology.orchestra.run.smithi064.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 717 kB/s | 36 kB 00:00 2024-03-25T16:27:59.375 INFO:teuthology.orchestra.run.smithi064.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-25T16:27:59.419 INFO:teuthology.orchestra.run.smithi086.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 232 kB/s | 24 kB 00:00 2024-03-25T16:27:59.461 INFO:teuthology.orchestra.run.smithi086.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 18 kB 00:00 2024-03-25T16:27:59.503 INFO:teuthology.orchestra.run.smithi086.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.8 MB/s | 174 kB 00:00 2024-03-25T16:27:59.509 INFO:teuthology.orchestra.run.smithi064.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-25T16:27:59.528 INFO:teuthology.orchestra.run.smithi086.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T16:27:59.554 INFO:teuthology.orchestra.run.smithi086.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:27:59.567 INFO:teuthology.orchestra.run.smithi064.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 343 kB/s | 20 kB 00:00 2024-03-25T16:27:59.579 INFO:teuthology.orchestra.run.smithi086.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 865 kB/s | 43 kB 00:00 2024-03-25T16:27:59.721 INFO:teuthology.orchestra.run.smithi086.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 193 kB/s | 32 kB 00:00 2024-03-25T16:27:59.746 INFO:teuthology.orchestra.run.smithi086.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 90 kB/s | 15 kB 00:00 2024-03-25T16:27:59.813 INFO:teuthology.orchestra.run.smithi086.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 2.2 MB/s | 203 kB 00:00 2024-03-25T16:27:59.872 INFO:teuthology.orchestra.run.smithi086.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 8.1 MB/s | 1.0 MB 00:00 2024-03-25T16:27:59.897 INFO:teuthology.orchestra.run.smithi086.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 712 kB/s | 59 kB 00:00 2024-03-25T16:27:59.922 INFO:teuthology.orchestra.run.smithi086.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-03-25T16:27:59.947 INFO:teuthology.orchestra.run.smithi086.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 82 kB/s | 52 kB 00:00 2024-03-25T16:27:59.950 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:59.950 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-25T16:27:59.951 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:27:59.963 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:27:59.964 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:28:00.052 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:28:00.053 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:28:00.575 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:28:00.727 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:28:00.790 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:28:00.853 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:28:00.882 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:28:00.924 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:28:01.018 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:28:01.125 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:28:01.176 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:28:01.894 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:28:01.947 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:28:01.981 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:28:02.030 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:28:02.059 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:28:02.526 INFO:teuthology.orchestra.run.smithi064.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 413 kB/s | 4.7 MB 00:11 2024-03-25T16:28:02.820 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:28:02.820 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:28:02.820 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:28:02.820 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:28:02.820 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:28:02.820 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:28:02.820 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:28:02.820 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:28:02.820 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:28:02.820 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:28:02.820 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:28:02.821 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:28:02.821 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:28:03.184 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:28:03.184 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:03.184 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:28:03.184 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:03.184 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:28:03.184 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:28:03.184 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:28:03.184 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:28:03.184 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:28:03.185 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:28:03.185 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:28:03.185 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:28:03.185 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:28:03.185 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:28:03.185 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:28:03.185 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:28:03.185 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:03.185 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:28:03.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:28:04.003 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:50 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:28:04.153 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:28:04.154 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-25T16:28:04.154 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-25T16:28:04.154 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-25T16:28:04.154 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:28:04.154 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:28:04.154 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-25T16:28:04.154 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:28:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:28:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:28:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:28:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:28:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:04.155 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:28:04.155 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-25T16:28:04.155 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-03-25T16:28:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:04.156 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 6.8 M 2024-03-25T16:28:04.156 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 33 M 2024-03-25T16:28:04.156 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:28:04.957 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-25T16:28:04.982 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 702 kB/s | 248 kB 00:00 2024-03-25T16:28:05.225 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 239 kB/s | 58 kB 00:00 2024-03-25T16:28:05.283 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 873 kB/s | 51 kB 00:00 2024-03-25T16:28:05.325 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-03-25T16:28:05.366 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 187 kB/s | 138 kB 00:00 2024-03-25T16:28:05.370 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:05.371 INFO:teuthology.orchestra.run.smithi086.stdout:Total 5.6 MB/s | 6.8 MB 00:01 2024-03-25T16:28:05.383 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:28:05.394 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:28:05.394 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:28:05.483 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:28:05.484 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:28:05.783 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:28:05.960 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:28:06.018 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:28:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:28:06.613 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:28:06.682 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:28:06.705 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:28:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:28:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:28:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:28:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:28:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:28:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:28:07.771 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:28:07.772 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:07.772 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:28:07.772 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:07.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:28:07.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:28:07.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:28:07.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:28:07.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:28:07.772 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:07.772 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:28:08.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-fuse 2024-03-25T16:28:08.583 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:03:54 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:28:08.735 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:28:08.735 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:08.735 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-25T16:28:08.735 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:08.735 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:28:08.735 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:28:08.735 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-25T16:28:08.735 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:28:08.736 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:08.736 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:28:08.736 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:08.736 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2024-03-25T16:28:08.736 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:08.736 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 964 k 2024-03-25T16:28:08.736 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 2.8 M 2024-03-25T16:28:08.736 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:28:08.851 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 689 kB/s | 79 kB 00:00 2024-03-25T16:28:11.634 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 305 kB/s | 885 kB 00:02 2024-03-25T16:28:11.635 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:11.636 INFO:teuthology.orchestra.run.smithi086.stdout:Total 332 kB/s | 964 kB 00:02 2024-03-25T16:28:11.636 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:28:11.659 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:28:11.659 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:28:11.733 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:28:11.733 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:28:11.912 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:28:12.052 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:28:12.127 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:28:12.838 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:28:12.838 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:28:13.137 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:28:13.137 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:13.138 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:28:13.138 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:28:13.138 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:13.138 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:28:13.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-volume 2024-03-25T16:28:14.012 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:04:00 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:28:14.161 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:28:14.161 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:14.161 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-25T16:28:14.161 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:14.161 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:28:14.161 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:28:14.162 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-25T16:28:14.162 INFO:teuthology.orchestra.run.smithi086.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:28:14.162 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:14.162 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:28:14.162 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:14.162 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2024-03-25T16:28:14.162 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:14.163 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 332 k 2024-03-25T16:28:14.163 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 1.5 M 2024-03-25T16:28:14.163 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:28:14.321 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 433 kB/s | 69 kB 00:00 2024-03-25T16:28:15.063 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 293 kB/s | 264 kB 00:00 2024-03-25T16:28:15.065 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:15.065 INFO:teuthology.orchestra.run.smithi086.stdout:Total 368 kB/s | 332 kB 00:00 2024-03-25T16:28:15.065 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:28:15.074 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:28:15.075 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:28:15.118 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:28:15.118 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:28:15.268 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:28:15.453 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:28:15.491 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:28:15.506 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:28:15.506 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:15.506 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:28:15.506 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:16.201 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:28:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:28:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:16.540 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:28:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:28:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:16.540 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:28:16.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados-devel 2024-03-25T16:28:17.467 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:04:03 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:28:17.614 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:28:17.615 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:17.615 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T16:28:17.615 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:17.615 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:28:17.616 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:28:17.616 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:17.616 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:28:17.616 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:17.616 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T16:28:17.616 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:17.616 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 127 k 2024-03-25T16:28:17.616 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 456 k 2024-03-25T16:28:17.616 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:28:18.752 INFO:teuthology.orchestra.run.smithi086.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 112 kB/s | 127 kB 00:01 2024-03-25T16:28:18.752 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:18.752 INFO:teuthology.orchestra.run.smithi086.stdout:Total 111 kB/s | 127 kB 00:01 2024-03-25T16:28:18.753 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:28:18.759 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:28:18.759 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:28:18.818 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:28:18.819 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:28:18.971 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:28:19.082 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:19.525 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:19.838 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:19.838 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:19.838 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:28:19.839 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:19.839 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:19.839 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:28:20.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs2 2024-03-25T16:28:20.666 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:04:06 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:28:20.766 INFO:teuthology.orchestra.run.smithi086.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:20.821 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:28:20.822 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-25T16:28:20.822 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:28:20.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs-devel 2024-03-25T16:28:21.266 INFO:teuthology.orchestra.run.smithi064.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 647 kB/s | 17 MB 00:26 2024-03-25T16:28:21.271 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:21.271 INFO:teuthology.orchestra.run.smithi064.stdout:Total 971 kB/s | 30 MB 00:31 2024-03-25T16:28:21.295 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:28:21.342 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:28:21.342 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:28:21.459 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:04:07 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:28:21.610 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:28:21.611 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:21.611 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T16:28:21.611 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:21.611 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:28:21.612 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:28:21.612 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:21.612 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:28:21.612 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:21.612 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T16:28:21.612 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:21.612 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 31 k 2024-03-25T16:28:21.612 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 140 k 2024-03-25T16:28:21.612 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:28:21.756 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:28:21.756 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:28:22.465 INFO:teuthology.orchestra.run.smithi086.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 37 kB/s | 31 kB 00:00 2024-03-25T16:28:22.465 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:22.465 INFO:teuthology.orchestra.run.smithi086.stdout:Total 36 kB/s | 31 kB 00:00 2024-03-25T16:28:22.466 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:28:22.469 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:28:22.469 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:28:22.507 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:28:22.507 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:28:22.640 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:28:22.746 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:23.096 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:28:23.121 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:23.169 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:28:23.214 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:28:23.246 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:28:23.319 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:28:23.362 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:28:23.394 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:28:23.403 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:23.403 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:23.403 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:28:23.403 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:23.403 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:23.403 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:28:23.491 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:28:23.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados2 2024-03-25T16:28:23.567 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:28:23.614 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:28:23.688 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:28:23.741 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:28:23.777 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:28:23.837 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:28:23.901 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:28:24.095 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:28:24.116 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:04:10 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:28:24.148 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:28:24.196 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:28:24.213 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:24.231 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:28:24.260 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:28:24.264 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:28:24.265 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-25T16:28:24.265 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:28:24.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librbd1 2024-03-25T16:28:24.339 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:28:24.371 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:28:24.406 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:28:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:28:24.479 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:28:24.514 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:28:24.606 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:28:24.645 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:28:24.721 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:28:24.868 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:04:10 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:28:24.870 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:28:24.965 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:28:25.017 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:28:25.018 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-25T16:28:25.018 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:28:25.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rados 2024-03-25T16:28:25.160 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:28:25.208 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:28:25.233 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:28:25.233 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:25.233 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:28:25.233 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:28:25.233 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:28:25.233 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:25.629 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:04:11 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:28:25.729 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:25.784 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:28:25.785 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-25T16:28:25.785 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:28:25.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rgw 2024-03-25T16:28:26.423 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:04:12 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:28:26.522 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:26.576 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:28:26.577 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-25T16:28:26.577 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:28:26.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-cephfs 2024-03-25T16:28:26.885 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:28:26.910 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:28:26.911 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:26.911 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:28:26.911 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:28:26.911 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:28:26.911 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:27.190 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:04:13 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:28:27.286 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:27.338 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:28:27.339 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-25T16:28:27.339 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:28:27.377 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:28:27.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rbd 2024-03-25T16:28:27.400 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:28:27.400 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:27.400 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:28:27.400 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:28:27.400 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:28:27.400 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:27.619 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:28:27.642 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:28:27.642 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:27.642 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:28:27.642 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:28:27.642 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:28:27.642 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:27.710 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:28:27.946 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:04:13 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:28:28.043 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:28.095 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:28:28.096 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-25T16:28:28.096 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:28:28.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-fuse 2024-03-25T16:28:28.712 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:04:14 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:28:28.860 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:28:28.861 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:28.861 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-03-25T16:28:28.861 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:28.861 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:28:28.861 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:28:28.861 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:28.861 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:28:28.861 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:28.861 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T16:28:28.861 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:28.861 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 85 k 2024-03-25T16:28:28.861 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 231 k 2024-03-25T16:28:28.861 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:28:29.278 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:28:29.279 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:28:29.279 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:28:29.279 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:28:29.279 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:28:29.279 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:28:29.279 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:28:29.279 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:28:29.279 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:28:29.279 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:28:29.279 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:28:29.279 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:28:29.279 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:28:29.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:28:29.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:28:29.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:28:29.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:28:29.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:28:29.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:28:29.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:28:29.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:28:29.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:28:29.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:28:29.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:28:29.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:28:29.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:28:29.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:28:29.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:28:29.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:28:29.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:28:29.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:28:29.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:28:29.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:28:29.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:28:29.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:28:29.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:28:29.768 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 94 kB/s | 85 kB 00:00 2024-03-25T16:28:29.768 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:29.768 INFO:teuthology.orchestra.run.smithi086.stdout:Total 94 kB/s | 85 kB 00:00 2024-03-25T16:28:29.769 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:28:29.776 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:28:29.776 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:28:29.833 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:28:29.834 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:28:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:28:30.075 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:30.257 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:28:30.257 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:30.257 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:28:30.257 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:30.257 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:30.257 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:30.257 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:30.257 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:30.257 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:30.258 INFO:teuthology.orchestra.run.smithi064.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:28:30.258 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:28:30.258 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:28:30.258 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:28:30.258 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:28:30.258 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:28:30.258 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:28:30.258 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:28:30.258 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:28:30.258 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:28:30.258 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:28:30.258 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:28:30.258 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:28:30.258 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:28:30.259 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:28:30.259 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:28:30.259 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:28:30.259 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:28:30.259 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:28:30.259 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:28:30.259 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:28:30.259 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:28:30.259 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:28:30.259 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:28:30.259 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:28:30.259 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:28:30.259 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:28:30.259 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:28:30.259 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:28:30.260 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:28:30.260 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:30.260 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:28:30.492 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:30.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-base 2024-03-25T16:28:30.765 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:30.766 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:30.766 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:28:30.766 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:30.766 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:30.766 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:28:30.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror 2024-03-25T16:28:31.198 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:04:08 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:28:31.298 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:31.352 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:28:31.353 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-25T16:28:31.354 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:28:31.412 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install cephadm 2024-03-25T16:28:31.559 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:04:17 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:28:31.709 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:28:31.710 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:31.710 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T16:28:31.710 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:31.710 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:28:31.710 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:28:31.710 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:31.710 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:28:31.710 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:31.711 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T16:28:31.711 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:31.712 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.1 M 2024-03-25T16:28:31.712 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 13 M 2024-03-25T16:28:31.712 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:28:31.983 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:04:08 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:28:32.138 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:28:32.139 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:28:32.139 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-25T16:28:32.139 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:28:32.139 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:28:32.139 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:28:32.139 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:32.139 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:28:32.140 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:28:32.140 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T16:28:32.140 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:32.140 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 747 k 2024-03-25T16:28:32.140 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 754 k 2024-03-25T16:28:32.140 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:28:35.827 INFO:teuthology.orchestra.run.smithi064.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 203 kB/s | 747 kB 00:03 2024-03-25T16:28:35.827 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:35.827 INFO:teuthology.orchestra.run.smithi064.stdout:Total 203 kB/s | 747 kB 00:03 2024-03-25T16:28:35.828 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:28:35.834 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:28:35.835 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:28:35.841 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:28:35.841 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:28:35.915 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:28:36.363 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:28:36.442 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:28:36.842 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:28:37.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:28:37.108 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:37.108 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:28:37.108 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:37.108 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:37.108 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:28:37.306 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:28:37.884 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:04:14 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:28:38.041 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:28:38.042 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:28:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T16:28:38.042 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:28:38.042 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:28:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:28:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:38.042 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:28:38.042 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:28:38.042 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T16:28:38.043 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:38.043 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 145 k 2024-03-25T16:28:38.043 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 436 k 2024-03-25T16:28:38.043 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:28:39.074 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 141 kB/s | 145 kB 00:01 2024-03-25T16:28:39.075 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:39.075 INFO:teuthology.orchestra.run.smithi064.stdout:Total 141 kB/s | 145 kB 00:01 2024-03-25T16:28:39.076 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:28:39.086 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:28:39.086 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:28:39.144 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:28:39.145 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:28:39.297 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:28:39.367 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:28:39.392 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:28:39.392 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:39.392 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:28:39.392 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:28:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:40.424 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:28:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:40.425 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:28:40.670 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr 2024-03-25T16:28:41.246 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:04:18 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:28:41.346 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:41.401 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:28:41.402 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-25T16:28:41.402 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:28:41.463 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:28:42.035 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:04:19 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:28:42.195 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:28:42.196 INFO:teuthology.orchestra.run.smithi064.stdout:====================================================================================== 2024-03-25T16:28:42.196 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-25T16:28:42.197 INFO:teuthology.orchestra.run.smithi064.stdout:====================================================================================== 2024-03-25T16:28:42.197 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:28:42.197 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:28:42.197 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-25T16:28:42.197 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:28:42.197 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:28:42.197 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:28:42.197 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:28:42.197 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:28:42.197 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:28:42.197 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:28:42.197 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:28:42.198 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:28:42.198 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:28:42.198 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:28:42.198 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:42.198 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:28:42.198 INFO:teuthology.orchestra.run.smithi064.stdout:====================================================================================== 2024-03-25T16:28:42.198 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-03-25T16:28:42.198 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:42.199 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 6.9 M 2024-03-25T16:28:42.199 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 26 M 2024-03-25T16:28:42.199 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:28:42.987 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 72 kB/s | 31 kB 00:00 2024-03-25T16:28:43.112 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 27 kB/s | 15 kB 00:00 2024-03-25T16:28:43.154 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-25T16:28:43.179 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 3.5 MB/s | 238 kB 00:00 2024-03-25T16:28:43.479 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.8 MB/s | 548 kB 00:00 2024-03-25T16:28:43.538 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 332 kB/s | 19 kB 00:00 2024-03-25T16:28:43.688 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.6 MB/s | 863 kB 00:00 2024-03-25T16:28:43.730 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 11 MB/s | 2.0 MB 00:00 2024-03-25T16:28:43.789 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 528 kB/s | 31 kB 00:00 2024-03-25T16:28:43.856 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-25T16:28:43.923 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 617 kB/s | 144 kB 00:00 2024-03-25T16:28:44.536 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 247 kB/s | 3.1 MB 00:12 2024-03-25T16:28:44.537 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:44.537 INFO:teuthology.orchestra.run.smithi086.stdout:Total 246 kB/s | 3.1 MB 00:12 2024-03-25T16:28:44.538 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:28:44.548 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:28:44.548 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:28:44.624 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:28:44.625 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:28:45.057 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:28:45.112 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:45.136 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:45.136 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:45.136 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:28:45.136 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:28:45.136 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:28:45.136 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:46.185 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:46.185 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:46.185 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:28:46.185 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:46.185 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:46.185 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:28:46.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd 2024-03-25T16:28:47.024 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:04:33 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:28:47.189 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:28:47.189 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:47.189 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-03-25T16:28:47.190 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:47.190 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:28:47.190 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:28:47.190 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:47.190 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:28:47.190 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:47.190 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T16:28:47.190 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:47.190 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 171 k 2024-03-25T16:28:47.190 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 494 k 2024-03-25T16:28:47.190 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:28:47.997 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 213 kB/s | 171 kB 00:00 2024-03-25T16:28:47.997 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:47.998 INFO:teuthology.orchestra.run.smithi086.stdout:Total 212 kB/s | 171 kB 00:00 2024-03-25T16:28:47.998 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:28:48.007 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:28:48.007 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:28:48.066 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:28:48.067 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:28:48.217 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:28:48.321 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:48.372 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 313 kB/s | 1.8 MB 00:05 2024-03-25T16:28:48.376 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:48.376 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.1 MB/s | 6.9 MB 00:06 2024-03-25T16:28:48.396 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:28:48.417 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:28:48.417 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:28:48.591 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:28:48.591 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:28:48.739 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:49.051 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:49.051 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:49.051 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:28:49.051 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:49.051 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:49.051 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:28:49.118 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:28:49.190 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:28:49.225 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:28:49.273 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:28:49.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:28:49.303 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:28:49.396 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:28:49.525 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:28:49.617 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:28:49.665 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:28:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:28:49.725 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:28:49.865 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:04:35 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:28:49.980 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:28:50.008 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:28:50.018 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:28:50.018 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:50.019 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T16:28:50.019 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:50.019 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:28:50.019 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:28:50.019 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-25T16:28:50.019 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:28:50.019 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:50.019 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:28:50.019 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:28:50.019 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2024-03-25T16:28:50.019 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:28:50.020 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 91 M 2024-03-25T16:28:50.020 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 498 M 2024-03-25T16:28:50.020 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:28:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:28:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:28:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:28:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:28:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:28:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:28:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:28:50.777 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:28:50.777 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:28:50.777 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:28:50.777 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:28:50.777 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:28:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:28:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:51.140 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:28:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:28:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:28:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:28:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:28:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:28:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:28:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:28:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:28:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:28:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:51.141 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:28:51.381 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:28:51.949 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:04:28 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:28:52.106 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:28:52.107 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================ 2024-03-25T16:28:52.107 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-25T16:28:52.107 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================ 2024-03-25T16:28:52.107 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:28:52.107 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:28:52.108 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-25T16:28:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:28:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:28:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:28:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:28:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:28:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:28:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:28:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:28:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:28:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:28:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:28:52.108 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:52.109 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:28:52.109 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================ 2024-03-25T16:28:52.109 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-03-25T16:28:52.109 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:28:52.109 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 37 M 2024-03-25T16:28:52.109 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 209 M 2024-03-25T16:28:52.109 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:28:52.490 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 209 kB/s | 32 kB 00:00 2024-03-25T16:28:52.512 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 691 kB/s | 17 kB 00:00 2024-03-25T16:28:52.554 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 820 kB/s | 34 kB 00:00 2024-03-25T16:28:52.763 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2024-03-25T16:28:52.930 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-25T16:28:52.972 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2024-03-25T16:28:53.030 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.3 MB/s | 3.0 MB 00:00 2024-03-25T16:28:53.414 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-25T16:28:53.864 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 966 kB/s | 803 kB 00:00 2024-03-25T16:28:53.931 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 378 kB/s | 195 kB 00:00 2024-03-25T16:28:54.048 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-25T16:29:08.880 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 457 kB/s | 7.4 MB 00:16 2024-03-25T16:29:08.882 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:08.883 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.2 MB/s | 37 MB 00:16 2024-03-25T16:29:08.925 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:29:08.951 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:29:08.951 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:29:09.329 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:29:09.329 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:29:10.053 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:29:10.128 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:29:10.159 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:29:10.256 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:29:10.296 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:29:10.480 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:29:10.503 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:29:10.582 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:29:11.254 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:29:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:29:12.665 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:29:13.748 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:29:13.777 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:29:16.317 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:29:16.317 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:29:16.317 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:29:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:29:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:29:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:29:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:29:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:29:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:29:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:29:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:29:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:29:16.690 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:29:16.690 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:16.690 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:29:16.690 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:29:16.690 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:29:16.690 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:29:16.690 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:29:16.690 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:29:16.691 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:29:16.691 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:29:16.691 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:29:16.691 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:29:16.691 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:29:16.691 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:29:16.691 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:29:16.691 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:16.691 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:29:17.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:29:17.649 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:04:54 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:29:17.805 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:29:17.808 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-25T16:29:17.808 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-25T16:29:17.808 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-25T16:29:17.808 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:29:17.808 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:29:17.808 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-25T16:29:17.808 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:29:17.808 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:29:17.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:29:17.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:29:17.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:29:17.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:29:17.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:29:17.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:29:17.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:29:17.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:29:17.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:29:17.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:29:17.809 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:17.809 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:29:17.810 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-25T16:29:17.810 INFO:teuthology.orchestra.run.smithi064.stdout:Install 13 Packages 2024-03-25T16:29:17.810 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:17.810 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 2.0 M 2024-03-25T16:29:17.810 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 26 M 2024-03-25T16:29:17.810 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:29:18.352 INFO:teuthology.orchestra.run.smithi064.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 169 kB/s | 24 kB 00:00 2024-03-25T16:29:18.377 INFO:teuthology.orchestra.run.smithi064.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 107 kB/s | 18 kB 00:00 2024-03-25T16:29:18.444 INFO:teuthology.orchestra.run.smithi064.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-25T16:29:18.469 INFO:teuthology.orchestra.run.smithi064.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-25T16:29:18.486 INFO:teuthology.orchestra.run.smithi064.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-03-25T16:29:18.511 INFO:teuthology.orchestra.run.smithi064.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-25T16:29:18.561 INFO:teuthology.orchestra.run.smithi064.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 301 kB/s | 15 kB 00:00 2024-03-25T16:29:18.587 INFO:teuthology.orchestra.run.smithi064.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-03-25T16:29:18.629 INFO:teuthology.orchestra.run.smithi064.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 3.0 MB/s | 203 kB 00:00 2024-03-25T16:29:18.662 INFO:teuthology.orchestra.run.smithi064.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-03-25T16:29:18.696 INFO:teuthology.orchestra.run.smithi064.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-03-25T16:29:18.738 INFO:teuthology.orchestra.run.smithi064.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-03-25T16:29:18.754 INFO:teuthology.orchestra.run.smithi064.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 95 kB/s | 52 kB 00:00 2024-03-25T16:29:18.756 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:18.756 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-25T16:29:18.757 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:29:18.770 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:29:18.771 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:29:18.868 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:29:18.868 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:29:19.388 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:29:19.569 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:29:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:29:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:29:19.724 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:29:19.766 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:29:19.863 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:29:19.972 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:29:20.026 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:29:20.789 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:29:20.830 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:29:20.864 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:29:20.913 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:29:20.942 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:29:21.659 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:29:21.659 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:29:21.659 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:29:21.659 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:29:21.659 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:29:21.659 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:29:21.659 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:29:21.660 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:29:21.660 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:29:21.660 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:29:21.660 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:29:21.660 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:29:21.660 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:29:21.995 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:29:21.995 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:21.995 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:29:21.995 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:29:21.996 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:29:21.996 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:29:21.996 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:29:21.996 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:29:21.996 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:29:21.996 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:29:21.996 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:29:21.996 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:29:21.996 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:29:21.996 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:29:21.996 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:29:21.996 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:29:21.997 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:21.997 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:29:22.243 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:29:22.818 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:04:59 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:29:22.976 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:29:22.977 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-25T16:29:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-25T16:29:22.977 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-25T16:29:22.977 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:29:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:29:22.977 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-25T16:29:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:29:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:29:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:29:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:29:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:29:22.977 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:22.977 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:29:22.978 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-25T16:29:22.978 INFO:teuthology.orchestra.run.smithi064.stdout:Install 6 Packages 2024-03-25T16:29:22.978 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:22.978 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 6.8 M 2024-03-25T16:29:22.978 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 33 M 2024-03-25T16:29:22.978 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:29:23.481 INFO:teuthology.orchestra.run.smithi064.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-25T16:29:23.507 INFO:teuthology.orchestra.run.smithi064.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 694 kB/s | 248 kB 00:00 2024-03-25T16:29:23.848 INFO:teuthology.orchestra.run.smithi064.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 169 kB/s | 58 kB 00:00 2024-03-25T16:29:24.099 INFO:teuthology.orchestra.run.smithi064.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 888 kB/s | 548 kB 00:00 2024-03-25T16:29:24.190 INFO:teuthology.orchestra.run.smithi064.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 149 kB/s | 51 kB 00:00 2024-03-25T16:29:24.208 INFO:teuthology.orchestra.run.smithi064.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 130 kB/s | 138 kB 00:01 2024-03-25T16:29:24.209 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:24.209 INFO:teuthology.orchestra.run.smithi064.stdout:Total 5.5 MB/s | 6.8 MB 00:01 2024-03-25T16:29:24.223 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:29:24.234 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:29:24.234 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:29:24.325 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:29:24.325 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:29:24.627 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:29:24.789 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:29:24.843 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:29:25.397 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:29:25.474 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:29:25.542 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:29:25.571 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:29:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:29:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:29:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:29:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:29:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:29:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:29:26.617 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:29:26.617 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:26.617 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:29:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:29:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:29:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:29:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:29:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:29:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:29:26.618 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:26.618 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:29:26.847 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-fuse 2024-03-25T16:29:27.428 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:05:04 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:29:27.585 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:29:27.585 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:27.586 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-25T16:29:27.586 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:27.586 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:29:27.586 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:29:27.586 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-25T16:29:27.586 INFO:teuthology.orchestra.run.smithi064.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:29:27.586 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:27.586 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:29:27.586 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:27.586 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-03-25T16:29:27.586 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:27.587 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 964 k 2024-03-25T16:29:27.587 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.8 M 2024-03-25T16:29:27.587 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:29:27.801 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 368 kB/s | 79 kB 00:00 2024-03-25T16:29:29.417 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 484 kB/s | 885 kB 00:01 2024-03-25T16:29:29.419 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:29.419 INFO:teuthology.orchestra.run.smithi064.stdout:Total 526 kB/s | 964 kB 00:01 2024-03-25T16:29:29.419 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:29:29.444 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:29:29.444 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:29:29.525 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:29:29.525 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:29:29.708 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:29:29.856 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:29:29.929 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:29:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:29:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:29:30.946 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:29:30.946 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:30.946 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:29:30.947 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:29:30.947 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:30.947 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:29:31.232 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-volume 2024-03-25T16:29:31.737 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 551 kB/s | 22 MB 00:41 2024-03-25T16:29:31.816 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:05:08 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:29:31.972 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:29:31.973 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:31.973 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-03-25T16:29:31.973 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:31.973 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:29:31.973 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:29:31.974 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-25T16:29:31.974 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:29:31.974 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:31.974 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:29:31.974 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:31.974 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-03-25T16:29:31.974 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:31.974 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 332 k 2024-03-25T16:29:31.974 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.5 M 2024-03-25T16:29:31.974 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:29:32.172 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 349 kB/s | 69 kB 00:00 2024-03-25T16:29:32.997 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 258 kB/s | 264 kB 00:01 2024-03-25T16:29:32.998 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:32.998 INFO:teuthology.orchestra.run.smithi064.stdout:Total 324 kB/s | 332 kB 00:01 2024-03-25T16:29:32.999 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:29:33.009 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:29:33.010 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:29:33.058 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:29:33.059 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:29:33.214 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:29:33.404 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:29:33.441 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:29:33.456 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:29:33.456 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:29:33.456 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:29:33.456 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:29:34.534 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:29:34.534 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:34.534 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:29:34.535 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:29:34.535 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:29:34.535 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:34.535 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:29:34.853 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados-devel 2024-03-25T16:29:35.436 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:29:35.589 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:29:35.590 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:35.590 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T16:29:35.590 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:35.590 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:29:35.591 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:29:35.591 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:35.591 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:29:35.591 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:35.591 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T16:29:35.591 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:35.591 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 127 k 2024-03-25T16:29:35.591 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 456 k 2024-03-25T16:29:35.591 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:29:36.494 INFO:teuthology.orchestra.run.smithi064.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 140 kB/s | 127 kB 00:00 2024-03-25T16:29:36.495 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:36.495 INFO:teuthology.orchestra.run.smithi064.stdout:Total 140 kB/s | 127 kB 00:00 2024-03-25T16:29:36.495 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:29:36.501 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:29:36.502 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:29:36.561 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:29:36.562 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:29:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:29:36.830 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:37.274 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:37.591 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:37.591 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:37.591 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:29:37.591 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:37.591 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:37.592 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:29:37.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs2 2024-03-25T16:29:38.396 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:05:15 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:29:38.496 INFO:teuthology.orchestra.run.smithi064.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:38.553 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:29:38.554 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-25T16:29:38.554 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:29:38.614 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs-devel 2024-03-25T16:29:39.193 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:05:16 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:29:39.347 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:29:39.348 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:39.348 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T16:29:39.348 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:39.348 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:29:39.348 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:29:39.349 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:39.349 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:29:39.349 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:39.349 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T16:29:39.349 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:39.349 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 31 k 2024-03-25T16:29:39.349 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 140 k 2024-03-25T16:29:39.349 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:29:39.808 INFO:teuthology.orchestra.run.smithi064.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 68 kB/s | 31 kB 00:00 2024-03-25T16:29:39.809 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:39.809 INFO:teuthology.orchestra.run.smithi064.stdout:Total 68 kB/s | 31 kB 00:00 2024-03-25T16:29:39.809 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:29:39.813 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:29:39.813 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:29:39.841 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:29:39.841 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:29:39.972 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:29:40.093 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:40.725 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:40.725 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:40.725 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:29:40.725 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:40.725 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:40.725 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:29:40.910 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados2 2024-03-25T16:29:41.490 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:05:18 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:29:41.589 INFO:teuthology.orchestra.run.smithi064.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:41.644 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:29:41.645 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-25T16:29:41.645 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:29:41.693 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librbd1 2024-03-25T16:29:42.273 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:05:19 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:29:42.373 INFO:teuthology.orchestra.run.smithi064.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:42.427 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:29:42.429 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-25T16:29:42.429 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:29:42.476 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rados 2024-03-25T16:29:43.052 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:05:20 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:29:43.152 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:43.207 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:29:43.208 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-25T16:29:43.208 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:29:43.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rgw 2024-03-25T16:29:43.824 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:05:20 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:29:43.923 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:43.979 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:29:43.981 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-25T16:29:43.981 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:29:44.020 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-cephfs 2024-03-25T16:29:44.596 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:05:21 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:29:44.695 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:44.749 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:29:44.750 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-25T16:29:44.751 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:29:44.793 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rbd 2024-03-25T16:29:45.387 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:05:22 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:29:45.490 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:45.545 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:29:45.546 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-25T16:29:45.547 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:29:45.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-fuse 2024-03-25T16:29:46.176 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:05:23 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:29:46.331 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:29:46.332 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:46.332 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-03-25T16:29:46.332 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:46.333 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:29:46.333 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:29:46.333 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:46.333 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:29:46.333 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:46.333 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T16:29:46.333 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:46.333 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 85 k 2024-03-25T16:29:46.333 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 231 k 2024-03-25T16:29:46.333 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:29:47.253 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 93 kB/s | 85 kB 00:00 2024-03-25T16:29:47.253 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:47.254 INFO:teuthology.orchestra.run.smithi064.stdout:Total 93 kB/s | 85 kB 00:00 2024-03-25T16:29:47.254 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:29:47.262 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:29:47.262 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:29:47.322 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:29:47.322 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:29:47.462 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:29:47.571 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:47.963 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:48.282 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:29:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:48.282 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:29:48.504 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-mirror 2024-03-25T16:29:49.083 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:05:26 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:29:49.238 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:29:49.239 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:49.239 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T16:29:49.239 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:49.239 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:29:49.239 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:29:49.239 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:49.239 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:29:49.240 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:49.240 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T16:29:49.240 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:49.240 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 3.1 M 2024-03-25T16:29:49.240 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 13 M 2024-03-25T16:29:49.240 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:29:56.372 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 443 kB/s | 3.1 MB 00:07 2024-03-25T16:29:56.372 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:56.373 INFO:teuthology.orchestra.run.smithi064.stdout:Total 443 kB/s | 3.1 MB 00:07 2024-03-25T16:29:56.373 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:29:56.384 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:29:56.384 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:29:56.464 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:29:56.464 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:29:56.912 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:29:56.989 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:57.013 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:57.013 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:29:57.014 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:29:57.014 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:29:57.014 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:29:57.014 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:58.231 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:58.232 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:58.232 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:29:58.232 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:58.232 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:58.232 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:29:58.480 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-nbd 2024-03-25T16:29:59.062 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:05:36 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:29:59.217 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:29:59.217 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-03-25T16:29:59.218 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:59.218 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:29:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:29:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:59.218 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:29:59.218 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:29:59.218 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T16:29:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:29:59.218 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 171 k 2024-03-25T16:29:59.218 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 494 k 2024-03-25T16:29:59.218 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:29:59.966 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 229 kB/s | 171 kB 00:00 2024-03-25T16:29:59.967 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:59.967 INFO:teuthology.orchestra.run.smithi064.stdout:Total 229 kB/s | 171 kB 00:00 2024-03-25T16:29:59.967 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:29:59.976 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:29:59.976 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:30:00.035 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:30:00.035 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:30:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:30:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:30:00.740 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:30:01.023 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:30:01.024 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:30:01.024 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:30:01.024 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:01.024 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:30:01.024 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:30:01.244 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:30:01.821 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:05:38 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:30:01.975 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:30:01.976 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:30:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T16:30:01.976 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:30:01.976 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:30:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:30:01.976 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-25T16:30:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:30:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:30:01.976 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:30:01.976 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:30:01.976 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-03-25T16:30:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:30:01.977 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 91 M 2024-03-25T16:30:01.977 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 498 M 2024-03-25T16:30:01.977 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:30:49.224 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 487 kB/s | 22 MB 00:47 2024-03-25T16:31:24.609 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 451 kB/s | 68 MB 02:34 2024-03-25T16:31:24.610 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:31:24.610 INFO:teuthology.orchestra.run.smithi086.stdout:Total 600 kB/s | 91 MB 02:34 2024-03-25T16:31:24.611 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:31:24.615 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:31:24.615 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:31:25.202 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:31:25.202 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:31:28.307 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:31:35.502 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:31:35.634 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:31:38.879 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:31:38.879 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:31:39.142 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:31:39.142 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:31:39.142 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:31:39.142 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:39.142 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:39.142 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:31:39.143 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:31:39.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:31:39.946 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:07:25 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:31:40.095 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:31:40.096 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:31:40.096 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T16:31:40.096 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:31:40.096 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:31:40.096 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:31:40.096 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:31:40.097 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:31:40.097 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:31:40.097 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T16:31:40.097 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:31:40.097 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 416 M 2024-03-25T16:31:40.097 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 1.7 G 2024-03-25T16:31:40.097 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:33:35.122 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 327 kB/s | 68 MB 03:33 2024-03-25T16:33:35.122 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:35.123 INFO:teuthology.orchestra.run.smithi064.stdout:Total 435 kB/s | 91 MB 03:33 2024-03-25T16:33:35.123 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:33:35.127 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:33:35.127 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:33:35.738 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:33:35.738 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:33:39.009 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:33:46.529 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:33:46.600 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:33:49.920 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:33:49.920 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:33:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:33:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:33:50.191 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:33:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:33:50.191 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:33:50.399 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:33:50.976 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:09:27 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:33:51.130 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:33:51.131 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:33:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T16:33:51.131 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:33:51.132 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:33:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:33:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:33:51.132 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:33:51.132 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:33:51.132 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T16:33:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:33:51.132 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 416 M 2024-03-25T16:33:51.132 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.7 G 2024-03-25T16:33:51.132 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:52:59.544 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 371 kB/s | 416 MB 19:08 2024-03-25T16:52:59.555 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:52:59.555 INFO:teuthology.orchestra.run.smithi064.stdout:Total 371 kB/s | 416 MB 19:08 2024-03-25T16:52:59.555 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:52:59.555 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:52:59.555 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:53:02.135 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:53:02.135 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:53:47.083 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:53:47.470 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:53:54.398 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:53:54.659 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:53:54.659 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:53:54.659 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:53:54.659 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:53:54.659 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:53:54.659 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:53:54.885 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:53:55.462 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:29:32 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:53:55.616 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:53:55.616 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-25T16:53:55.616 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T16:53:55.616 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-25T16:53:55.616 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:53:55.616 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:53:55.617 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:53:55.617 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:53:55.617 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================== 2024-03-25T16:53:55.617 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T16:53:55.617 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:53:55.617 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 1.4 M 2024-03-25T16:53:55.617 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 6.4 M 2024-03-25T16:53:55.618 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:53:58.018 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 609 kB/s | 1.4 MB 00:02 2024-03-25T16:53:58.019 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:53:58.019 INFO:teuthology.orchestra.run.smithi064.stdout:Total 608 kB/s | 1.4 MB 00:02 2024-03-25T16:53:58.019 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T16:53:58.023 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T16:53:58.023 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T16:53:58.070 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T16:53:58.070 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T16:53:58.340 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T16:53:58.444 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:53:58.843 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:53:59.124 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:53:59.124 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:53:59.124 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T16:53:59.124 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:53:59.124 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:53:59.124 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T16:53:59.316 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:53:59.887 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:29:36 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T16:54:00.040 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T16:54:00.041 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:54:00.041 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T16:54:00.041 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:54:00.041 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T16:54:00.041 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:54:00.042 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:54:00.042 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T16:54:00.042 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T16:54:00.042 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T16:54:00.042 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T16:54:00.042 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 176 M 2024-03-25T16:54:00.042 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 753 M 2024-03-25T16:54:00.042 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T16:57:50.848 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 271 kB/s | 416 MB 26:10 2024-03-25T16:57:50.849 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:57:50.849 INFO:teuthology.orchestra.run.smithi086.stdout:Total 271 kB/s | 416 MB 26:10 2024-03-25T16:57:50.849 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:57:50.852 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:57:50.853 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:57:53.316 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:57:53.316 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:58:35.436 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:58:35.535 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:58:43.229 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:58:43.494 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:58:43.495 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:58:43.495 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:58:43.495 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:58:43.495 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:58:43.495 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:58:43.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:58:44.286 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:34:30 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:58:44.435 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:58:44.435 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-25T16:58:44.435 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T16:58:44.435 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-25T16:58:44.435 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:58:44.436 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:58:44.436 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:58:44.436 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:58:44.436 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-25T16:58:44.436 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T16:58:44.436 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:58:44.436 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 1.4 M 2024-03-25T16:58:44.436 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 6.4 M 2024-03-25T16:58:44.437 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T16:58:48.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 363 kB/s | 1.4 MB 00:04 2024-03-25T16:58:48.465 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:58:48.465 INFO:teuthology.orchestra.run.smithi086.stdout:Total 363 kB/s | 1.4 MB 00:04 2024-03-25T16:58:48.465 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T16:58:48.470 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T16:58:48.470 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T16:58:48.514 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T16:58:48.515 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T16:58:48.769 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T16:58:48.878 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:58:49.258 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:58:49.533 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:58:49.534 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:58:49.534 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T16:58:49.534 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:58:49.534 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:58:49.534 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T16:58:49.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:58:50.287 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:34:36 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T16:58:50.434 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T16:58:50.435 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:58:50.435 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T16:58:50.436 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:58:50.436 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T16:58:50.436 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:58:50.436 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:58:50.436 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T16:58:50.436 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T16:58:50.436 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T16:58:50.436 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T16:58:50.436 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 176 M 2024-03-25T16:58:50.436 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 753 M 2024-03-25T16:58:50.436 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T17:03:40.055 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 310 kB/s | 176 MB 09:40 2024-03-25T17:03:40.056 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:03:40.056 INFO:teuthology.orchestra.run.smithi064.stdout:Total 310 kB/s | 176 MB 09:40 2024-03-25T17:03:40.056 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T17:03:40.060 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T17:03:40.060 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T17:03:41.170 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T17:03:41.170 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T17:04:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T17:04:00.373 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:04:04.866 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:04:05.135 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:04:05.135 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:04:05.135 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T17:04:05.136 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:04:05.136 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:04:05.136 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T17:04:05.326 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:04:05.898 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:39:42 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T17:04:06.052 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T17:04:06.052 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:04:06.053 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T17:04:06.053 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:04:06.053 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T17:04:06.053 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:04:06.053 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:04:06.053 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T17:04:06.053 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:04:06.053 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T17:04:06.053 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:04:06.054 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 672 M 2024-03-25T17:04:06.054 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 3.1 G 2024-03-25T17:04:06.054 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T17:08:53.085 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 298 kB/s | 176 MB 10:02 2024-03-25T17:08:53.085 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:08:53.085 INFO:teuthology.orchestra.run.smithi086.stdout:Total 298 kB/s | 176 MB 10:02 2024-03-25T17:08:53.086 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T17:08:53.088 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T17:08:53.089 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T17:08:54.144 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T17:08:54.145 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T17:09:12.087 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T17:09:12.296 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:09:16.915 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:09:17.182 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:09:17.182 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:09:17.182 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T17:09:17.182 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:09:17.182 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:09:17.182 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T17:09:17.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:09:17.928 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:45:03 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T17:09:18.077 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T17:09:18.078 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:09:18.078 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T17:09:18.078 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:09:18.078 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T17:09:18.078 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:09:18.078 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:09:18.078 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T17:09:18.078 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:09:18.079 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T17:09:18.079 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:09:18.079 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 672 M 2024-03-25T17:09:18.079 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 3.1 G 2024-03-25T17:09:18.079 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T17:28:31.870 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 470 kB/s | 672 MB 24:25 2024-03-25T17:28:31.901 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:28:31.902 INFO:teuthology.orchestra.run.smithi064.stdout:Total 470 kB/s | 672 MB 24:25 2024-03-25T17:28:31.902 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T17:28:31.902 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T17:28:31.902 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T17:28:36.030 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T17:28:36.030 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T17:29:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T17:29:51.317 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:29:58.963 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:29:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:29:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:29:59.224 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T17:29:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:29:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:29:59.224 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T17:29:59.556 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:30:00.134 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 1:05:37 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T17:30:00.234 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:30:00.289 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T17:30:00.290 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-03-25T17:30:00.290 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T17:30:00.351 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:30:00.928 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 1:05:37 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T17:30:01.082 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T17:30:01.083 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:30:01.083 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T17:30:01.084 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:30:01.084 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T17:30:01.084 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:30:01.084 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:30:01.084 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T17:30:01.084 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:30:01.084 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T17:30:01.084 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:30:01.084 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 20 M 2024-03-25T17:30:01.084 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 103 M 2024-03-25T17:30:01.084 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T17:31:23.233 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 246 kB/s | 20 MB 01:22 2024-03-25T17:31:23.234 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:31:23.234 INFO:teuthology.orchestra.run.smithi064.stdout:Total 246 kB/s | 20 MB 01:22 2024-03-25T17:31:23.235 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T17:31:23.238 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T17:31:23.238 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T17:31:23.395 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T17:31:23.395 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T17:31:25.750 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T17:31:25.868 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:31:26.787 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:31:27.053 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:31:27.053 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:31:27.053 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T17:31:27.053 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:31:27.053 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:31:27.054 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T17:31:27.225 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:31:27.799 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 1:07:04 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T17:31:27.955 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T17:31:27.956 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:31:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T17:31:27.956 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:31:27.956 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T17:31:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:31:27.956 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:31:27.956 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T17:31:27.956 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:31:27.957 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T17:31:27.957 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:31:27.957 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 48 M 2024-03-25T17:31:27.957 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 173 M 2024-03-25T17:31:27.957 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T17:33:40.188 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 370 kB/s | 48 MB 02:12 2024-03-25T17:33:40.209 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:33:40.209 INFO:teuthology.orchestra.run.smithi064.stdout:Total 370 kB/s | 48 MB 02:12 2024-03-25T17:33:40.209 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T17:33:40.209 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T17:33:40.209 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T17:33:40.519 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T17:33:40.520 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T17:33:45.539 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T17:33:45.639 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:33:47.113 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:33:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:33:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:33:47.384 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T17:33:47.384 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:33:47.385 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:33:47.385 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T17:33:47.549 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:33:48.125 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 1:09:25 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T17:33:48.281 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T17:33:48.282 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:33:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T17:33:48.282 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:33:48.282 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T17:33:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:33:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:33:48.282 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T17:33:48.282 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:33:48.283 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T17:33:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:33:48.283 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 99 M 2024-03-25T17:33:48.283 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 449 M 2024-03-25T17:33:48.283 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T17:35:42.868 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 434 kB/s | 672 MB 26:24 2024-03-25T17:35:42.869 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:35:42.869 INFO:teuthology.orchestra.run.smithi086.stdout:Total 434 kB/s | 672 MB 26:24 2024-03-25T17:35:42.869 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T17:35:42.874 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T17:35:42.874 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T17:35:46.843 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T17:35:46.844 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T17:36:57.737 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T17:36:57.903 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:37:06.946 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:37:07.209 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:37:07.210 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:37:07.210 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T17:37:07.210 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:37:07.210 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:37:07.210 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T17:37:07.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:37:08.066 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 1:12:54 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T17:37:08.162 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:37:08.213 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T17:37:08.214 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-25T17:37:08.215 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T17:37:08.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:37:08.857 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 1:12:54 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T17:37:09.007 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T17:37:09.007 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:37:09.008 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T17:37:09.008 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:37:09.008 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T17:37:09.008 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:37:09.008 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:37:09.008 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T17:37:09.008 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:37:09.008 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T17:37:09.008 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:37:09.009 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 20 M 2024-03-25T17:37:09.009 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 103 M 2024-03-25T17:37:09.009 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T17:37:47.369 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 526 kB/s | 20 MB 00:38 2024-03-25T17:37:47.370 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:37:47.370 INFO:teuthology.orchestra.run.smithi086.stdout:Total 526 kB/s | 20 MB 00:38 2024-03-25T17:37:47.370 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T17:37:47.374 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T17:37:47.374 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T17:37:47.523 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T17:37:47.523 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T17:37:49.743 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T17:37:49.917 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:50.875 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:51.139 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:51.139 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:37:51.139 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T17:37:51.139 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:37:51.140 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:37:51.140 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T17:37:51.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:37:51.905 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 1:13:37 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T17:37:52.052 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T17:37:52.053 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:37:52.053 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T17:37:52.053 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:37:52.053 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T17:37:52.054 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:37:52.054 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:37:52.054 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T17:37:52.054 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:37:52.054 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T17:37:52.054 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:37:52.054 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 48 M 2024-03-25T17:37:52.054 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 173 M 2024-03-25T17:37:52.054 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T17:38:26.549 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 363 kB/s | 99 MB 04:38 2024-03-25T17:38:26.550 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:38:26.550 INFO:teuthology.orchestra.run.smithi064.stdout:Total 363 kB/s | 99 MB 04:38 2024-03-25T17:38:26.550 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T17:38:26.553 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T17:38:26.554 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T17:38:27.192 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T17:38:27.192 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T17:38:38.093 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T17:38:38.199 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:41.074 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:41.337 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:41.337 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:38:41.337 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T17:38:41.337 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:38:41.337 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:38:41.337 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T17:38:41.492 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:38:42.067 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 1:14:19 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T17:38:42.222 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T17:38:42.223 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:38:42.223 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T17:38:42.223 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:38:42.223 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T17:38:42.223 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:38:42.223 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:38:42.223 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T17:38:42.223 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:38:42.223 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T17:38:42.223 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:38:42.224 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 287 M 2024-03-25T17:38:42.224 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.2 G 2024-03-25T17:38:42.224 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T17:39:21.320 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 547 kB/s | 48 MB 01:29 2024-03-25T17:39:21.321 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:39:21.321 INFO:teuthology.orchestra.run.smithi086.stdout:Total 547 kB/s | 48 MB 01:29 2024-03-25T17:39:21.322 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T17:39:21.325 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T17:39:21.325 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T17:39:21.639 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T17:39:21.639 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T17:39:26.362 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T17:39:26.451 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:39:27.725 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:39:27.982 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:39:27.982 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:39:27.982 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T17:39:27.982 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:39:27.982 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:39:27.982 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T17:39:28.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:39:28.725 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 1:15:14 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T17:39:28.873 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T17:39:28.873 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:39:28.873 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T17:39:28.873 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:39:28.873 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T17:39:28.874 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:39:28.874 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:39:28.874 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T17:39:28.874 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:39:28.874 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T17:39:28.874 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:39:28.874 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 99 M 2024-03-25T17:39:28.874 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 449 M 2024-03-25T17:39:28.875 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T17:44:38.289 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 326 kB/s | 99 MB 05:09 2024-03-25T17:44:38.290 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:44:38.290 INFO:teuthology.orchestra.run.smithi086.stdout:Total 326 kB/s | 99 MB 05:09 2024-03-25T17:44:38.290 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T17:44:38.293 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T17:44:38.294 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T17:44:38.905 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T17:44:38.906 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T17:44:49.196 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T17:44:49.312 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:52.112 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:52.375 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:52.375 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:44:52.375 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T17:44:52.375 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:44:52.375 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:44:52.375 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T17:44:52.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:44:53.136 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 1:20:39 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T17:44:53.285 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T17:44:53.285 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:44:53.285 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T17:44:53.285 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:44:53.285 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T17:44:53.285 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:44:53.285 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:44:53.286 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T17:44:53.286 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:44:53.286 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T17:44:53.286 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:44:53.286 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 287 M 2024-03-25T17:44:53.286 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 1.2 G 2024-03-25T17:44:53.287 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T17:54:50.759 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 303 kB/s | 287 MB 16:08 2024-03-25T17:54:50.759 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:50.759 INFO:teuthology.orchestra.run.smithi064.stdout:Total 303 kB/s | 287 MB 16:08 2024-03-25T17:54:50.759 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T17:54:50.762 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T17:54:50.762 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T17:54:52.551 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T17:54:52.552 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T17:55:23.677 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T17:55:23.790 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:28.018 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:28.314 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:28.314 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:55:28.314 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T17:55:28.314 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:28.314 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:55:28.315 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T17:55:28.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:55:29.086 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 1:31:06 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T17:55:29.240 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T17:55:29.240 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:55:29.240 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T17:55:29.241 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:55:29.241 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T17:55:29.241 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:55:29.241 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:55:29.241 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T17:55:29.241 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:55:29.241 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T17:55:29.241 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:55:29.241 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 5.9 M 2024-03-25T17:55:29.241 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 28 M 2024-03-25T17:55:29.241 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T17:55:43.648 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 422 kB/s | 5.9 MB 00:14 2024-03-25T17:55:43.649 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:43.649 INFO:teuthology.orchestra.run.smithi064.stdout:Total 422 kB/s | 5.9 MB 00:14 2024-03-25T17:55:43.649 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T17:55:43.653 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T17:55:43.654 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T17:55:43.729 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T17:55:43.729 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T17:55:44.510 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T17:55:44.625 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:45.211 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:45.481 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:45.481 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:55:45.481 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T17:55:45.481 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:45.481 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:55:45.481 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T17:55:45.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:55:46.236 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 1:31:23 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T17:55:46.390 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T17:55:46.391 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:55:46.391 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T17:55:46.391 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:55:46.391 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T17:55:46.392 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:55:46.392 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:55:46.392 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T17:55:46.392 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:55:46.392 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T17:55:46.392 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:55:46.392 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 109 k 2024-03-25T17:55:46.392 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 440 k 2024-03-25T17:55:46.392 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T17:55:47.842 INFO:teuthology.orchestra.run.smithi064.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 75 kB/s | 109 kB 00:01 2024-03-25T17:55:47.842 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:47.843 INFO:teuthology.orchestra.run.smithi064.stdout:Total 75 kB/s | 109 kB 00:01 2024-03-25T17:55:47.843 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T17:55:47.847 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T17:55:47.847 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T17:55:47.886 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T17:55:47.887 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T17:55:48.012 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T17:55:48.119 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:55:48.496 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:55:48.753 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:55:48.753 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:55:48.753 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T17:55:48.753 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:48.753 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:55:48.753 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T17:55:48.914 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:55:49.491 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 1:31:26 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T17:55:49.646 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T17:55:49.647 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:55:49.647 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T17:55:49.647 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:55:49.647 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T17:55:49.648 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:55:49.648 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:55:49.648 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T17:55:49.648 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:55:49.648 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T17:55:49.648 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:55:49.648 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 5.4 M 2024-03-25T17:55:49.648 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 25 M 2024-03-25T17:55:49.648 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T17:56:00.869 INFO:teuthology.orchestra.run.smithi064.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 488 kB/s | 5.4 MB 00:11 2024-03-25T17:56:00.870 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:00.870 INFO:teuthology.orchestra.run.smithi064.stdout:Total 488 kB/s | 5.4 MB 00:11 2024-03-25T17:56:00.871 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T17:56:00.875 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T17:56:00.875 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T17:56:00.945 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T17:56:00.946 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T17:56:01.654 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T17:56:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:56:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:56:02.644 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:56:02.644 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:56:02.644 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T17:56:02.644 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:02.644 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:56:02.644 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T17:56:02.842 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados2-debuginfo 2024-03-25T17:56:03.418 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 1:31:40 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T17:56:03.574 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T17:56:03.574 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:56:03.574 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T17:56:03.574 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:56:03.574 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T17:56:03.575 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:56:03.575 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:56:03.575 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T17:56:03.575 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:56:03.575 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T17:56:03.575 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:56:03.575 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 44 M 2024-03-25T17:56:03.575 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 253 M 2024-03-25T17:56:03.575 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T17:58:08.042 INFO:teuthology.orchestra.run.smithi064.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 364 kB/s | 44 MB 02:04 2024-03-25T17:58:08.043 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:08.043 INFO:teuthology.orchestra.run.smithi064.stdout:Total 364 kB/s | 44 MB 02:04 2024-03-25T17:58:08.043 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T17:58:08.046 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T17:58:08.046 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T17:58:08.360 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T17:58:08.360 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T17:58:13.711 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T17:58:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:58:15.610 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:58:15.877 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:58:15.877 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:58:15.877 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T17:58:15.877 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:15.877 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:58:15.877 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T17:58:16.057 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:58:16.631 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 1:33:53 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T17:58:16.788 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T17:58:16.789 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:58:16.789 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T17:58:16.789 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:58:16.789 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T17:58:16.789 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:58:16.789 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:58:16.789 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T17:58:16.789 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T17:58:16.789 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T17:58:16.790 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T17:58:16.790 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 52 M 2024-03-25T17:58:16.790 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 308 M 2024-03-25T17:58:16.790 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T17:58:19.060 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 364 kB/s | 287 MB 13:25 2024-03-25T17:58:19.060 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:19.060 INFO:teuthology.orchestra.run.smithi086.stdout:Total 364 kB/s | 287 MB 13:25 2024-03-25T17:58:19.061 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T17:58:19.064 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T17:58:19.064 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T17:58:20.773 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T17:58:20.773 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T17:58:49.985 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T17:58:50.081 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:57.334 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:57.609 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:57.610 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:58:57.610 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T17:58:57.610 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:57.610 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:58:57.610 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T17:58:57.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:58:58.392 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 1:34:44 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T17:58:58.541 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T17:58:58.542 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:58:58.542 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T17:58:58.543 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:58:58.543 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T17:58:58.543 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:58:58.543 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:58:58.543 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T17:58:58.543 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:58:58.543 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T17:58:58.543 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:58:58.543 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 5.9 M 2024-03-25T17:58:58.543 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 28 M 2024-03-25T17:58:58.543 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T17:59:15.191 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 366 kB/s | 5.9 MB 00:16 2024-03-25T17:59:15.192 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:15.192 INFO:teuthology.orchestra.run.smithi086.stdout:Total 365 kB/s | 5.9 MB 00:16 2024-03-25T17:59:15.193 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T17:59:15.197 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T17:59:15.197 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T17:59:15.267 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T17:59:15.268 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T17:59:16.002 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T17:59:16.106 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:59:16.710 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:59:16.972 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:59:16.972 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:59:16.972 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T17:59:16.972 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:16.972 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:59:16.972 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T17:59:17.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:59:17.720 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 1:35:03 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T17:59:17.869 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T17:59:17.870 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:59:17.870 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T17:59:17.870 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:59:17.870 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T17:59:17.870 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:59:17.870 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:59:17.870 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T17:59:17.871 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:59:17.871 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T17:59:17.871 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:59:17.871 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 109 k 2024-03-25T17:59:17.871 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 440 k 2024-03-25T17:59:17.871 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T17:59:19.052 INFO:teuthology.orchestra.run.smithi086.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 92 kB/s | 109 kB 00:01 2024-03-25T17:59:19.053 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:19.053 INFO:teuthology.orchestra.run.smithi086.stdout:Total 92 kB/s | 109 kB 00:01 2024-03-25T17:59:19.053 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T17:59:19.057 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T17:59:19.057 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T17:59:19.095 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T17:59:19.095 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T17:59:19.212 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T17:59:19.326 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:59:19.700 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:59:19.960 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:59:19.961 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:59:19.961 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T17:59:19.961 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:19.961 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:59:19.961 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T17:59:20.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:59:20.719 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 1:35:06 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T17:59:20.867 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T17:59:20.868 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:59:20.868 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T17:59:20.869 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:59:20.869 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T17:59:20.869 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:59:20.869 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:59:20.869 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T17:59:20.869 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:59:20.870 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T17:59:20.870 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:59:20.870 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 5.4 M 2024-03-25T17:59:20.870 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 25 M 2024-03-25T17:59:20.870 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T17:59:36.279 INFO:teuthology.orchestra.run.smithi086.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 356 kB/s | 5.4 MB 00:15 2024-03-25T17:59:36.280 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:36.280 INFO:teuthology.orchestra.run.smithi086.stdout:Total 356 kB/s | 5.4 MB 00:15 2024-03-25T17:59:36.280 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T17:59:36.284 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T17:59:36.285 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T17:59:36.352 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T17:59:36.352 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T17:59:37.018 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T17:59:37.135 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:59:37.662 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:59:37.926 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:59:37.927 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:59:37.927 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T17:59:37.927 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:37.927 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:59:37.927 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T17:59:38.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados2-debuginfo 2024-03-25T17:59:38.683 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 1:35:24 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T17:59:38.831 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T17:59:38.831 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:59:38.832 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T17:59:38.832 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:59:38.832 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T17:59:38.832 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:59:38.832 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:59:38.832 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T17:59:38.832 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T17:59:38.832 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T17:59:38.832 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T17:59:38.832 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 44 M 2024-03-25T17:59:38.832 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 253 M 2024-03-25T17:59:38.832 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T18:01:21.708 INFO:teuthology.orchestra.run.smithi064.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 287 kB/s | 52 MB 03:04 2024-03-25T18:01:21.709 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:21.709 INFO:teuthology.orchestra.run.smithi064.stdout:Total 287 kB/s | 52 MB 03:04 2024-03-25T18:01:21.709 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T18:01:21.713 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T18:01:21.713 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T18:01:22.072 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T18:01:22.072 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T18:01:28.417 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T18:01:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:01:30.776 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:01:31.051 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:01:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:31.052 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T18:01:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:31.052 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T18:01:31.350 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:01:31.922 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 1:37:08 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T18:01:32.077 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T18:01:32.078 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:01:32.078 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T18:01:32.078 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:01:32.079 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T18:01:32.079 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:01:32.079 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:32.079 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T18:01:32.079 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:01:32.079 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T18:01:32.079 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:32.079 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 615 k 2024-03-25T18:01:32.079 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.1 M 2024-03-25T18:01:32.079 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T18:01:34.827 INFO:teuthology.orchestra.run.smithi064.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 224 kB/s | 615 kB 00:02 2024-03-25T18:01:34.827 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:34.828 INFO:teuthology.orchestra.run.smithi064.stdout:Total 224 kB/s | 615 kB 00:02 2024-03-25T18:01:34.828 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T18:01:34.832 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T18:01:34.832 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T18:01:34.877 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T18:01:34.878 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T18:01:35.053 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T18:01:35.171 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:01:35.587 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:01:35.850 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:01:35.850 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:35.850 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T18:01:35.850 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:35.850 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:35.850 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T18:01:36.069 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:01:36.644 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 1:37:13 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T18:01:36.800 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T18:01:36.801 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:01:36.801 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T18:01:36.801 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:01:36.801 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T18:01:36.801 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:01:36.801 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:36.801 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T18:01:36.801 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:01:36.801 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T18:01:36.801 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:36.802 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 771 k 2024-03-25T18:01:36.802 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.7 M 2024-03-25T18:01:36.802 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T18:01:40.263 INFO:teuthology.orchestra.run.smithi064.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 223 kB/s | 771 kB 00:03 2024-03-25T18:01:40.263 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:40.264 INFO:teuthology.orchestra.run.smithi064.stdout:Total 223 kB/s | 771 kB 00:03 2024-03-25T18:01:40.264 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T18:01:40.268 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T18:01:40.268 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T18:01:40.312 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T18:01:40.312 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T18:01:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T18:01:40.624 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:01:40.989 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:01:41.275 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:01:41.275 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:41.275 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T18:01:41.275 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:41.276 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:41.276 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T18:01:41.473 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:01:42.045 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 1:37:19 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T18:01:42.200 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T18:01:42.201 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:01:42.201 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T18:01:42.201 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:01:42.201 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T18:01:42.201 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:01:42.201 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:42.201 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T18:01:42.201 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:01:42.202 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T18:01:42.202 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:42.202 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 1.1 M 2024-03-25T18:01:42.202 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 4.3 M 2024-03-25T18:01:42.202 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T18:01:46.149 INFO:teuthology.orchestra.run.smithi064.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 292 kB/s | 1.1 MB 00:03 2024-03-25T18:01:46.150 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:46.150 INFO:teuthology.orchestra.run.smithi064.stdout:Total 291 kB/s | 1.1 MB 00:03 2024-03-25T18:01:46.150 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T18:01:46.154 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T18:01:46.155 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T18:01:46.198 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T18:01:46.198 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T18:01:46.429 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T18:01:46.535 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:01:46.912 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:01:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:01:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:47.173 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T18:01:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:47.173 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T18:01:47.385 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:01:47.960 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 1:37:24 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T18:01:48.117 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T18:01:48.118 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:01:48.118 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T18:01:48.118 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:01:48.118 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T18:01:48.118 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:01:48.118 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:48.118 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T18:01:48.118 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:01:48.118 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T18:01:48.118 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:48.119 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 337 k 2024-03-25T18:01:48.119 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.0 M 2024-03-25T18:01:48.119 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T18:01:50.227 INFO:teuthology.orchestra.run.smithi064.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 160 kB/s | 337 kB 00:02 2024-03-25T18:01:50.228 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:50.228 INFO:teuthology.orchestra.run.smithi064.stdout:Total 160 kB/s | 337 kB 00:02 2024-03-25T18:01:50.228 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T18:01:50.232 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T18:01:50.232 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T18:01:50.273 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T18:01:50.274 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T18:01:50.420 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T18:01:50.521 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:01:50.876 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:01:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:01:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:51.139 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T18:01:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:51.139 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T18:01:51.326 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:01:51.900 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 1:37:28 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T18:01:52.055 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T18:01:52.056 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:01:52.056 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T18:01:52.056 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:01:52.056 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T18:01:52.056 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:01:52.056 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:52.057 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T18:01:52.057 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:01:52.057 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T18:01:52.057 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:52.057 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 509 k 2024-03-25T18:01:52.057 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.5 M 2024-03-25T18:01:52.057 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T18:01:53.397 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 380 kB/s | 509 kB 00:01 2024-03-25T18:01:53.398 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:53.398 INFO:teuthology.orchestra.run.smithi064.stdout:Total 379 kB/s | 509 kB 00:01 2024-03-25T18:01:53.398 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T18:01:53.402 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T18:01:53.402 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T18:01:53.444 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T18:01:53.444 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T18:01:53.611 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T18:01:53.715 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:01:54.098 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:01:54.372 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:01:54.372 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:54.372 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T18:01:54.372 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:54.372 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:54.372 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T18:01:54.555 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:01:55.131 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 1:37:32 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T18:01:55.287 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T18:01:55.287 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:01:55.288 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T18:01:55.288 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:01:55.288 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T18:01:55.288 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:01:55.288 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:55.288 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T18:01:55.288 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:01:55.288 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T18:01:55.288 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:01:55.289 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 52 M 2024-03-25T18:01:55.289 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 298 M 2024-03-25T18:01:55.289 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T18:02:25.073 INFO:teuthology.orchestra.run.smithi086.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 273 kB/s | 44 MB 02:46 2024-03-25T18:02:25.073 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:02:25.074 INFO:teuthology.orchestra.run.smithi086.stdout:Total 273 kB/s | 44 MB 02:46 2024-03-25T18:02:25.074 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T18:02:25.077 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T18:02:25.078 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T18:02:25.375 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T18:02:25.376 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T18:02:30.405 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T18:02:30.496 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:02:32.290 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:02:32.553 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:02:32.553 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:02:32.553 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T18:02:32.553 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:32.553 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:02:32.553 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T18:02:32.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:02:33.357 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 1:38:19 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T18:02:33.508 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T18:02:33.508 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:02:33.509 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T18:02:33.509 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:02:33.509 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T18:02:33.509 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:02:33.509 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:02:33.509 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T18:02:33.509 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:02:33.509 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T18:02:33.509 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:02:33.510 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 52 M 2024-03-25T18:02:33.510 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 308 M 2024-03-25T18:02:33.510 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T18:04:59.210 INFO:teuthology.orchestra.run.smithi086.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 364 kB/s | 52 MB 02:25 2024-03-25T18:04:59.211 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:59.211 INFO:teuthology.orchestra.run.smithi086.stdout:Total 364 kB/s | 52 MB 02:25 2024-03-25T18:04:59.211 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T18:04:59.214 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T18:04:59.214 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T18:04:59.557 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T18:04:59.557 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T18:05:05.522 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T18:05:05.666 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:07.776 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:08.033 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:08.033 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:08.033 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T18:05:08.033 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:08.033 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:08.033 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T18:05:08.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:05:08.813 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 1:40:54 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T18:05:08.971 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T18:05:08.972 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:05:08.972 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T18:05:08.972 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:05:08.972 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T18:05:08.972 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:05:08.972 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:08.972 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T18:05:08.972 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:05:08.972 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T18:05:08.973 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:08.973 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 615 k 2024-03-25T18:05:08.973 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 2.1 M 2024-03-25T18:05:08.973 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T18:05:10.887 INFO:teuthology.orchestra.run.smithi086.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 321 kB/s | 615 kB 00:01 2024-03-25T18:05:10.888 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:10.888 INFO:teuthology.orchestra.run.smithi086.stdout:Total 321 kB/s | 615 kB 00:01 2024-03-25T18:05:10.888 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T18:05:10.892 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T18:05:10.892 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T18:05:10.934 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T18:05:10.934 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T18:05:11.101 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T18:05:11.202 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:05:11.611 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:05:11.867 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:05:11.867 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:11.867 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T18:05:11.867 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:11.867 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:11.867 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T18:05:11.895 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 268 kB/s | 52 MB 03:16 2024-03-25T18:05:11.896 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:11.896 INFO:teuthology.orchestra.run.smithi064.stdout:Total 268 kB/s | 52 MB 03:16 2024-03-25T18:05:11.896 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T18:05:11.900 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T18:05:11.900 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T18:05:12.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:05:12.250 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T18:05:12.250 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T18:05:12.639 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 1:40:58 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T18:05:12.789 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T18:05:12.790 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:05:12.790 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T18:05:12.790 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:05:12.790 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T18:05:12.790 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:05:12.790 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:12.790 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T18:05:12.790 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:05:12.791 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T18:05:12.791 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:12.791 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 771 k 2024-03-25T18:05:12.791 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 2.7 M 2024-03-25T18:05:12.791 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T18:05:15.340 INFO:teuthology.orchestra.run.smithi086.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 303 kB/s | 771 kB 00:02 2024-03-25T18:05:15.340 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:15.340 INFO:teuthology.orchestra.run.smithi086.stdout:Total 302 kB/s | 771 kB 00:02 2024-03-25T18:05:15.341 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T18:05:15.345 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T18:05:15.345 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T18:05:15.386 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T18:05:15.386 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T18:05:15.566 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T18:05:15.664 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:05:16.039 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:05:16.300 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:05:16.300 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:16.300 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T18:05:16.300 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:16.300 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:16.300 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T18:05:16.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:05:17.082 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 1:41:03 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T18:05:17.230 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T18:05:17.231 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:05:17.231 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T18:05:17.231 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:05:17.231 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T18:05:17.231 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:05:17.231 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:17.231 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T18:05:17.231 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:05:17.231 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T18:05:17.232 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:17.232 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 1.1 M 2024-03-25T18:05:17.232 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 4.3 M 2024-03-25T18:05:17.232 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T18:05:18.512 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T18:05:18.605 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:05:20.690 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:05:20.962 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:05:20.962 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:05:20.962 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T18:05:20.963 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:20.963 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:05:20.963 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T18:05:21.153 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:05:21.730 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 1:40:58 ago on Mon 25 Mar 2024 04:24:23 PM UTC. 2024-03-25T18:05:21.887 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-25T18:05:21.888 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:05:21.888 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-03-25T18:05:21.888 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:05:21.888 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-25T18:05:21.888 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:05:21.888 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:05:21.888 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-25T18:05:21.888 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-25T18:05:21.888 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-25T18:05:21.888 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:05:21.889 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 1.3 M 2024-03-25T18:05:21.889 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 5.5 M 2024-03-25T18:05:21.889 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-25T18:05:23.134 INFO:teuthology.orchestra.run.smithi086.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 195 kB/s | 1.1 MB 00:05 2024-03-25T18:05:23.135 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:23.136 INFO:teuthology.orchestra.run.smithi086.stdout:Total 195 kB/s | 1.1 MB 00:05 2024-03-25T18:05:23.136 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T18:05:23.140 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T18:05:23.140 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T18:05:23.184 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T18:05:23.184 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T18:05:23.402 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T18:05:23.508 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:05:23.932 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:05:24.189 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:05:24.189 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:24.190 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T18:05:24.190 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:24.190 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:24.190 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T18:05:24.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:05:24.962 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 1:41:10 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T18:05:25.111 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T18:05:25.112 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:05:25.112 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T18:05:25.113 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:05:25.113 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T18:05:25.113 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:05:25.113 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:25.113 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T18:05:25.113 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:05:25.113 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T18:05:25.113 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:25.113 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 337 k 2024-03-25T18:05:25.113 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 1.0 M 2024-03-25T18:05:25.113 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T18:05:26.805 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 261 kB/s | 1.3 MB 00:04 2024-03-25T18:05:26.806 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:26.806 INFO:teuthology.orchestra.run.smithi064.stdout:Total 261 kB/s | 1.3 MB 00:04 2024-03-25T18:05:26.807 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-25T18:05:26.811 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-25T18:05:26.811 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-25T18:05:26.857 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-25T18:05:26.857 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-25T18:05:27.104 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-25T18:05:27.216 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:27.412 INFO:teuthology.orchestra.run.smithi086.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 147 kB/s | 337 kB 00:02 2024-03-25T18:05:27.412 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:27.413 INFO:teuthology.orchestra.run.smithi086.stdout:Total 147 kB/s | 337 kB 00:02 2024-03-25T18:05:27.413 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T18:05:27.417 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T18:05:27.417 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T18:05:27.457 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T18:05:27.458 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T18:05:27.597 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T18:05:27.641 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:27.703 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:05:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:05:27.911 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-25T18:05:27.911 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:27.912 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:05:27.912 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-25T18:05:28.068 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:05:28.226 DEBUG:teuthology.parallel:result is None 2024-03-25T18:05:28.330 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:05:28.330 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:28.330 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T18:05:28.330 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:28.330 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:28.330 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T18:05:28.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:05:29.097 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 1:41:15 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T18:05:29.246 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T18:05:29.247 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:05:29.247 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T18:05:29.247 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:05:29.247 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T18:05:29.247 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:05:29.247 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:29.247 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T18:05:29.247 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:05:29.247 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T18:05:29.247 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:29.247 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 509 k 2024-03-25T18:05:29.247 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 2.5 M 2024-03-25T18:05:29.247 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T18:05:31.023 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 287 kB/s | 509 kB 00:01 2024-03-25T18:05:31.023 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:31.024 INFO:teuthology.orchestra.run.smithi086.stdout:Total 286 kB/s | 509 kB 00:01 2024-03-25T18:05:31.024 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T18:05:31.028 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T18:05:31.028 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T18:05:31.067 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T18:05:31.068 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T18:05:31.226 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T18:05:31.330 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:05:31.712 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:05:31.974 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:05:31.974 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:31.974 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T18:05:31.974 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:31.974 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:31.974 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T18:05:32.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:05:32.745 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 1:41:18 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T18:05:32.894 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T18:05:32.895 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:05:32.895 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T18:05:32.895 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:05:32.895 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T18:05:32.895 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:05:32.895 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:32.895 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T18:05:32.896 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:05:32.896 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T18:05:32.896 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:05:32.896 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 52 M 2024-03-25T18:05:32.896 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 298 M 2024-03-25T18:05:32.896 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T18:08:23.774 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 309 kB/s | 52 MB 02:50 2024-03-25T18:08:23.775 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:23.775 INFO:teuthology.orchestra.run.smithi086.stdout:Total 309 kB/s | 52 MB 02:50 2024-03-25T18:08:23.775 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T18:08:23.778 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T18:08:23.778 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T18:08:24.112 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T18:08:24.112 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T18:08:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T18:08:30.121 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:08:32.119 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:08:32.373 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:08:32.373 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:32.373 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T18:08:32.373 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:32.373 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:32.373 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T18:08:32.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:08:33.131 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 1:44:19 ago on Mon 25 Mar 2024 04:24:14 PM UTC. 2024-03-25T18:08:33.278 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-25T18:08:33.279 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:08:33.279 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-25T18:08:33.279 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:08:33.279 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-25T18:08:33.279 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:08:33.280 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:33.280 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-25T18:08:33.280 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-25T18:08:33.280 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-25T18:08:33.280 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:33.280 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 1.3 M 2024-03-25T18:08:33.280 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 5.5 M 2024-03-25T18:08:33.280 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-25T18:08:39.317 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 212 kB/s | 1.3 MB 00:06 2024-03-25T18:08:39.318 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:39.318 INFO:teuthology.orchestra.run.smithi086.stdout:Total 212 kB/s | 1.3 MB 00:06 2024-03-25T18:08:39.319 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-25T18:08:39.323 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-25T18:08:39.323 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-25T18:08:39.366 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-25T18:08:39.367 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-25T18:08:39.619 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-25T18:08:39.715 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:08:40.204 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:08:40.465 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:08:40.465 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:40.465 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-25T18:08:40.465 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:40.465 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:40.465 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-25T18:08:40.658 DEBUG:teuthology.parallel:result is None 2024-03-25T18:08:40.658 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:08:40.875 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:08:40.899 INFO:teuthology.orchestra.run.smithi064.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:08:40.900 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:08:40.900 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:08:40.902 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:08:41.114 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:08:41.135 INFO:teuthology.orchestra.run.smithi086.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:08:41.135 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:08:41.135 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:08:41.137 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:08:41.137 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:08:41.137 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:08:41.168 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T18:08:41.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:08:41.203 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:08:41.204 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:08:41.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:08:41.239 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:08:41.308 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T18:08:41.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:08:41.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:08:41.402 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:08:41.403 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:08:41.403 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:08:41.431 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:08:41.499 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T18:08:41.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:08:41.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:08:41.592 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:08:41.592 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:08:41.592 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:08:41.620 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:08:41.687 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T18:08:41.688 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:08:41.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:08:41.781 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:08:41.863 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:08:41.863 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /var/log/ceph 2024-03-25T18:08:41.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /var/log/ceph 2024-03-25T18:08:41.891 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:08:41.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:08:41.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:08:41.960 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:08:41.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:08:42.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:08:42.073 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:08:42.073 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:08:42.073 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621263', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621263', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi064.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOsESA1elaXn55kueK3fg/Dl/CEXysJyvHdNcX1qTpJ6zrBpeeQqkkwPdMgV3RBpENoBbyHZLR0nrbBUrKKPduo=', 'smithi086.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBh9+NaMT+DLhes+BYo8wW/byWZV2VoUHyX7ijMCUMoyWHexkOSxRdg1lbBDy8P4D9z1Hprw2BYNCbuEm1Uygsc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:08:42.073 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:08:42.076 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:08:42.090 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:08:42.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:08:42.157 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:08:42.157 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:08:42.210 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:08:42.210 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-03-25T18:08:42.265 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:08:42.265 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:08:42.265 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T18:08:42.265 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:08:42.265 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:08:42.265 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-25 16:29:57.248057773 +0000 2024-03-25T18:08:42.265 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-25 16:23:30.136558566 +0000 2024-03-25T18:08:42.265 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-25 16:23:30.136558566 +0000 2024-03-25T18:08:42.265 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-25 16:23:30.136558566 +0000 2024-03-25T18:08:42.265 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:08:42.330 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-25T18:08:42.330 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-25T18:08:42.330 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000252631 s, 2.0 MB/s 2024-03-25T18:08:42.332 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:08:42.387 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-03-25T18:08:42.441 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:08:42.442 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:08:42.442 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T18:08:42.442 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:08:42.442 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:08:42.442 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-25 16:29:57.248057773 +0000 2024-03-25T18:08:42.442 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-25 16:23:30.497551429 +0000 2024-03-25T18:08:42.442 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-25 16:23:30.497551429 +0000 2024-03-25T18:08:42.442 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-25 16:23:30.497551429 +0000 2024-03-25T18:08:42.442 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:08:42.507 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-25T18:08:42.507 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-25T18:08:42.507 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000186763 s, 2.7 MB/s 2024-03-25T18:08:42.508 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:08:42.563 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-03-25T18:08:42.617 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:08:42.617 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:08:42.617 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T18:08:42.617 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:08:42.617 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:08:42.618 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-25 16:29:57.248057773 +0000 2024-03-25T18:08:42.618 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-25 16:23:30.839544667 +0000 2024-03-25T18:08:42.618 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-25 16:23:30.839544667 +0000 2024-03-25T18:08:42.618 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-25 16:23:30.839544667 +0000 2024-03-25T18:08:42.618 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:08:42.682 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-25T18:08:42.682 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-25T18:08:42.682 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000166327 s, 3.1 MB/s 2024-03-25T18:08:42.684 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:08:42.739 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-03-25T18:08:42.793 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:08:42.794 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:08:42.794 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T18:08:42.794 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:08:42.794 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:08:42.794 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-25 16:29:57.249057753 +0000 2024-03-25T18:08:42.794 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-25 16:23:31.186537806 +0000 2024-03-25T18:08:42.794 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-25 16:23:31.186537806 +0000 2024-03-25T18:08:42.794 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-25 16:23:31.186537806 +0000 2024-03-25T18:08:42.794 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:08:42.858 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-25T18:08:42.858 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-25T18:08:42.858 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000181022 s, 2.8 MB/s 2024-03-25T18:08:42.859 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:08:42.914 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:08:42.914 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T18:08:42.914 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:08:42.931 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:08:42.931 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2024-03-25T18:08:42.988 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:08:42.988 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:08:42.988 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T18:08:42.988 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:08:42.988 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:08:42.988 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-25 16:28:45.381099499 +0000 2024-03-25T18:08:42.988 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-25 16:23:18.239603213 +0000 2024-03-25T18:08:42.988 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-25 16:23:18.239603213 +0000 2024-03-25T18:08:42.988 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-25 16:23:18.239603213 +0000 2024-03-25T18:08:42.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:08:43.056 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-25T18:08:43.057 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-25T18:08:43.057 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000223862 s, 2.3 MB/s 2024-03-25T18:08:43.058 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:08:43.116 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2024-03-25T18:08:43.172 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:08:43.172 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:08:43.173 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T18:08:43.173 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:08:43.173 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:08:43.173 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-25 16:28:45.382099478 +0000 2024-03-25T18:08:43.173 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-25 16:23:18.590595495 +0000 2024-03-25T18:08:43.173 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-25 16:23:18.590595495 +0000 2024-03-25T18:08:43.173 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-25 16:23:18.590595495 +0000 2024-03-25T18:08:43.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:08:43.240 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-25T18:08:43.240 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-25T18:08:43.240 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000216348 s, 2.4 MB/s 2024-03-25T18:08:43.241 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:08:43.298 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-25 16:28:45.382099478 +0000 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-25 16:23:18.933587952 +0000 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-25 16:23:18.933587952 +0000 2024-03-25T18:08:43.354 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-25 16:23:18.933587952 +0000 2024-03-25T18:08:43.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:08:43.421 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-25T18:08:43.421 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-25T18:08:43.421 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000203034 s, 2.5 MB/s 2024-03-25T18:08:43.422 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:08:43.479 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2024-03-25T18:08:43.534 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:08:43.534 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:08:43.534 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T18:08:43.534 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:08:43.534 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:08:43.534 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-25 16:28:45.382099478 +0000 2024-03-25T18:08:43.534 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-25 16:23:19.270580542 +0000 2024-03-25T18:08:43.534 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-25 16:23:19.270580542 +0000 2024-03-25T18:08:43.534 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-25 16:23:19.270580542 +0000 2024-03-25T18:08:43.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:08:43.600 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-25T18:08:43.600 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-25T18:08:43.600 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000204518 s, 2.5 MB/s 2024-03-25T18:08:43.602 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:08:43.657 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:08:43.657 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:08:43.657 INFO:tasks.ceph:Generating config... 2024-03-25T18:08:43.659 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:08:43.659 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:08:43.659 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:08:43.659 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:08:43.659 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:08:43.659 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:08:43.659 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:08:43.659 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:08:43.659 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:08:43.659 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:08:43.659 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:08:43.659 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:08:43.659 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:08:43.659 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:08:43.659 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:08:43.659 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:08:43.660 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:08:43.660 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:08:43.660 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:08:43.660 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:08:43.660 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:08:43.660 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:08:43.660 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:08:43.660 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:08:43.660 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:08:43.660 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:08:43.660 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:08:43.660 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:08:43.660 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:08:43.660 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:08:43.660 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:08:43.660 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:08:43.660 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:08:43.661 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:08:43.661 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:08:43.661 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:08:43.661 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:08:43.661 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:08:43.661 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:08:43.661 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:08:43.661 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:08:43.661 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:08:43.661 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:08:43.661 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:08:43.661 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:08:43.661 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:08:43.661 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:08:43.661 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:08:43.661 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:08:43.662 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:08:43.662 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:08:43.662 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:08:43.662 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:08:43.662 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:08:43.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:08:43.718 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:08:43.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:08:43.767 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:08:43.794 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.64:6789]'), ('mon.c', '[v1:172.21.15.64:6790]'), ('mon.b', '[v1:172.21.15.86:6789]')] 2024-03-25T18:08:43.795 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.64:6789],[v1:172.21.15.64:6790],[v1:172.21.15.86:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:08:43.795 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:08:43.796 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:08:43.849 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.64:6789]' --addv c '[v1:172.21.15.64:6790]' --addv b '[v1:172.21.15.86:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:08:43.934 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:08:43.934 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: generated fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:08:43.934 INFO:teuthology.orchestra.run.smithi064.stdout:setting min_mon_release = quincy 2024-03-25T18:08:43.934 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 0 2024-03-25T18:08:43.934 INFO:teuthology.orchestra.run.smithi064.stdout:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:08:43.934 INFO:teuthology.orchestra.run.smithi064.stdout:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:08:43.934 INFO:teuthology.orchestra.run.smithi064.stdout:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:08:43.934 INFO:teuthology.orchestra.run.smithi064.stdout:min_mon_release 17 (quincy) 2024-03-25T18:08:43.934 INFO:teuthology.orchestra.run.smithi064.stdout:election_strategy: 3 2024-03-25T18:08:43.934 INFO:teuthology.orchestra.run.smithi064.stdout:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:08:43.934 INFO:teuthology.orchestra.run.smithi064.stdout:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:08:43.935 INFO:teuthology.orchestra.run.smithi064.stdout:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:08:43.935 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:08:43.936 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:08:43.950 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f3bd93ea-5bde-4b52-b027-4fe9ef2c973e... 2024-03-25T18:08:43.951 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:08:43.993 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:08:44.040 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-03-25T18:08:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: chdir = "" 2024-03-25T18:08:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:08:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: auth supported = cephx 2024-03-25T18:08:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: filestore xattr use omap = true 2024-03-25T18:08:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: mon clock drift allowed = 1.000 2024-03-25T18:08:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: osd crush chooseleaf type = 0 2024-03-25T18:08:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: auth debug = true 2024-03-25T18:08:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on old message = true 2024-03-25T18:08:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on bug = true 2024-03-25T18:08:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:08:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: mon pg warn max object skew = 0 2024-03-25T18:08:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:08:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:08:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default size = 2 2024-03-25T18:08:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow primary affinity = true 2024-03-25T18:08:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow pg remap = true 2024-03-25T18:08:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:08:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:08:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on no sortbitwise = false 2024-03-25T18:08:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:08:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on too few osds = false 2024-03-25T18:08:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:08:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:08:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: mon_allow_pool_size_one = true 2024-03-25T18:08:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:08:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: osd default data pool replay window = 5 2024-03-25T18:08:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: mon allow pool delete = true 2024-03-25T18:08:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: mon cluster log file level = debug 2024-03-25T18:08:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: debug asserts on shutdown = true 2024-03-25T18:08:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: mon health detail to clog = false 2024-03-25T18:08:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: mon host = "[v1:172.21.15.64:6789],[v1:172.21.15.64:6790],[v1:172.21.15.86:6789]" 2024-03-25T18:08:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: mon client directed command retry = 5 2024-03-25T18:08:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: mon election default strategy = 3 2024-03-25T18:08:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: ms bind msgr2 = False 2024-03-25T18:08:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: ms inject socket failures = 5000 2024-03-25T18:08:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: ms type = async 2024-03-25T18:08:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: osd heartbeat grace = 80 2024-03-25T18:08:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:08:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.043 INFO:teuthology.orchestra.run.smithi064.stdout:[osd] 2024-03-25T18:08:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: osd journal size = 100 2024-03-25T18:08:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub load threshold = 5.0 2024-03-25T18:08:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub max interval = 600 2024-03-25T18:08:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:08:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: osd recover clone overlap = true 2024-03-25T18:08:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: osd recovery max chunk = 1048576 2024-03-25T18:08:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug shutdown = true 2024-03-25T18:08:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug op order = true 2024-03-25T18:08:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify stray on activate = true 2024-03-25T18:08:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug trim objects = true 2024-03-25T18:08:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: osd open classes on start = true 2024-03-25T18:08:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug pg log writeout = true 2024-03-25T18:08:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:08:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: osd map max advance = 10 2024-03-25T18:08:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: journal zero on create = true 2024-03-25T18:08:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:08:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: filestore apply finisher threads = 3 2024-03-25T18:08:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: bdev debug aio = true 2024-03-25T18:08:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug misdirected ops = true 2024-03-25T18:08:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: bdev async discard = True 2024-03-25T18:08:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: bdev enable discard = True 2024-03-25T18:08:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore allocator = stupid 2024-03-25T18:08:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore block size = 96636764160 2024-03-25T18:08:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore fsck on mount = True 2024-03-25T18:08:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore zero block detection = True 2024-03-25T18:08:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluefs = 1 2024-03-25T18:08:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluestore = 1 2024-03-25T18:08:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: debug monc = 20 2024-03-25T18:08:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-03-25T18:08:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: debug osd = 20 2024-03-25T18:08:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: debug rocksdb = 10 2024-03-25T18:08:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:08:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd full ratio = 0.9 2024-03-25T18:08:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:08:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:08:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: osd class default list = * 2024-03-25T18:08:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: osd class load list = * 2024-03-25T18:08:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: osd client watch timeout = 120 2024-03-25T18:08:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify cached snaps = True 2024-03-25T18:08:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify missing on start = True 2024-03-25T18:08:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:08:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: osd fast shutdown = False 2024-03-25T18:08:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: osd max markdown count = 1000 2024-03-25T18:08:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock override recovery settings = True 2024-03-25T18:08:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: osd objectstore = bluestore 2024-03-25T18:08:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue = debug_random 2024-03-25T18:08:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue cut off = debug_random 2024-03-25T18:08:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: osd shutdown pgref assert = True 2024-03-25T18:08:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.047 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr] 2024-03-25T18:08:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-03-25T18:08:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: debug mgr = 20 2024-03-25T18:08:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-03-25T18:08:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-03-25T18:08:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:08:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:08:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: mgr/telemetry/nag = false 2024-03-25T18:08:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.047 INFO:teuthology.orchestra.run.smithi064.stdout:[mon] 2024-03-25T18:08:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-03-25T18:08:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-03-25T18:08:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: debug paxos = 20 2024-03-25T18:08:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-03-25T18:08:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: mon data avail warn = 5 2024-03-25T18:08:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: mon mgr mkfs grace = 240 2024-03-25T18:08:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:08:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd reporter subtree level = osd 2024-03-25T18:08:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd prime pg temp = true 2024-03-25T18:08:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:08:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:08:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:08:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:08:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:08:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:08:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:08:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: # 1m isn't quite enough 2024-03-25T18:08:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:08:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:08:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: mon min osdmap epochs = 50 2024-03-25T18:08:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd crush smoke test = False 2024-03-25T18:08:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune interval = 2 2024-03-25T18:08:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune min = 15 2024-03-25T18:08:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:08:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: mon scrub interval = 300 2024-03-25T18:08:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on pool no app = False 2024-03-25T18:08:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: paxos service trim min = 10 2024-03-25T18:08:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:08:44.050 INFO:teuthology.orchestra.run.smithi064.stdout:[client] 2024-03-25T18:08:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: rgw cache enabled = true 2024-03-25T18:08:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable ops log = true 2024-03-25T18:08:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable usage log = true 2024-03-25T18:08:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:08:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:08:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: debug monc = 20 2024-03-25T18:08:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-03-25T18:08:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: debug objecter = 20 2024-03-25T18:08:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: debug rados = 20 2024-03-25T18:08:44.050 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.a] 2024-03-25T18:08:44.050 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.c] 2024-03-25T18:08:44.050 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.b] 2024-03-25T18:08:44.051 INFO:teuthology.orchestra.run.smithi086.stdout:[global] 2024-03-25T18:08:44.051 INFO:teuthology.orchestra.run.smithi086.stdout: chdir = "" 2024-03-25T18:08:44.051 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:08:44.051 INFO:teuthology.orchestra.run.smithi086.stdout: auth supported = cephx 2024-03-25T18:08:44.051 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.051 INFO:teuthology.orchestra.run.smithi086.stdout: filestore xattr use omap = true 2024-03-25T18:08:44.051 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.051 INFO:teuthology.orchestra.run.smithi086.stdout: mon clock drift allowed = 1.000 2024-03-25T18:08:44.051 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.051 INFO:teuthology.orchestra.run.smithi086.stdout: osd crush chooseleaf type = 0 2024-03-25T18:08:44.051 INFO:teuthology.orchestra.run.smithi086.stdout: auth debug = true 2024-03-25T18:08:44.051 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.051 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on old message = true 2024-03-25T18:08:44.052 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on bug = true 2024-03-25T18:08:44.052 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.052 INFO:teuthology.orchestra.run.smithi086.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:08:44.052 INFO:teuthology.orchestra.run.smithi086.stdout: mon pg warn max object skew = 0 2024-03-25T18:08:44.052 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.052 INFO:teuthology.orchestra.run.smithi086.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:08:44.052 INFO:teuthology.orchestra.run.smithi086.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:08:44.052 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.052 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default size = 2 2024-03-25T18:08:44.052 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.052 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow primary affinity = true 2024-03-25T18:08:44.052 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow pg remap = true 2024-03-25T18:08:44.052 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:08:44.052 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:08:44.052 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on no sortbitwise = false 2024-03-25T18:08:44.053 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:08:44.053 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on too few osds = false 2024-03-25T18:08:44.053 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:08:44.053 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:08:44.053 INFO:teuthology.orchestra.run.smithi086.stdout: mon_allow_pool_size_one = true 2024-03-25T18:08:44.053 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.053 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:08:44.053 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.053 INFO:teuthology.orchestra.run.smithi086.stdout: osd default data pool replay window = 5 2024-03-25T18:08:44.053 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.053 INFO:teuthology.orchestra.run.smithi086.stdout: mon allow pool delete = true 2024-03-25T18:08:44.053 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.053 INFO:teuthology.orchestra.run.smithi086.stdout: mon cluster log file level = debug 2024-03-25T18:08:44.053 INFO:teuthology.orchestra.run.smithi086.stdout: debug asserts on shutdown = true 2024-03-25T18:08:44.053 INFO:teuthology.orchestra.run.smithi086.stdout: mon health detail to clog = false 2024-03-25T18:08:44.054 INFO:teuthology.orchestra.run.smithi086.stdout: mon host = "[v1:172.21.15.64:6789],[v1:172.21.15.64:6790],[v1:172.21.15.86:6789]" 2024-03-25T18:08:44.054 INFO:teuthology.orchestra.run.smithi086.stdout: mon client directed command retry = 5 2024-03-25T18:08:44.054 INFO:teuthology.orchestra.run.smithi086.stdout: mon election default strategy = 3 2024-03-25T18:08:44.054 INFO:teuthology.orchestra.run.smithi086.stdout: ms bind msgr2 = False 2024-03-25T18:08:44.054 INFO:teuthology.orchestra.run.smithi086.stdout: ms inject socket failures = 5000 2024-03-25T18:08:44.054 INFO:teuthology.orchestra.run.smithi086.stdout: ms type = async 2024-03-25T18:08:44.054 INFO:teuthology.orchestra.run.smithi086.stdout: osd heartbeat grace = 80 2024-03-25T18:08:44.054 INFO:teuthology.orchestra.run.smithi086.stdout: fsid = f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:08:44.054 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.054 INFO:teuthology.orchestra.run.smithi086.stdout:[osd] 2024-03-25T18:08:44.054 INFO:teuthology.orchestra.run.smithi086.stdout: osd journal size = 100 2024-03-25T18:08:44.054 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.054 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub load threshold = 5.0 2024-03-25T18:08:44.054 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub max interval = 600 2024-03-25T18:08:44.054 INFO:teuthology.orchestra.run.smithi086.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:08:44.055 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.055 INFO:teuthology.orchestra.run.smithi086.stdout: osd recover clone overlap = true 2024-03-25T18:08:44.055 INFO:teuthology.orchestra.run.smithi086.stdout: osd recovery max chunk = 1048576 2024-03-25T18:08:44.055 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.055 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug shutdown = true 2024-03-25T18:08:44.055 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug op order = true 2024-03-25T18:08:44.055 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify stray on activate = true 2024-03-25T18:08:44.055 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug trim objects = true 2024-03-25T18:08:44.055 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.055 INFO:teuthology.orchestra.run.smithi086.stdout: osd open classes on start = true 2024-03-25T18:08:44.055 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug pg log writeout = true 2024-03-25T18:08:44.055 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.055 INFO:teuthology.orchestra.run.smithi086.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:08:44.055 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.055 INFO:teuthology.orchestra.run.smithi086.stdout: osd map max advance = 10 2024-03-25T18:08:44.056 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.056 INFO:teuthology.orchestra.run.smithi086.stdout: journal zero on create = true 2024-03-25T18:08:44.056 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.056 INFO:teuthology.orchestra.run.smithi086.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:08:44.056 INFO:teuthology.orchestra.run.smithi086.stdout: filestore apply finisher threads = 3 2024-03-25T18:08:44.056 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.056 INFO:teuthology.orchestra.run.smithi086.stdout: bdev debug aio = true 2024-03-25T18:08:44.056 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug misdirected ops = true 2024-03-25T18:08:44.056 INFO:teuthology.orchestra.run.smithi086.stdout: bdev async discard = True 2024-03-25T18:08:44.056 INFO:teuthology.orchestra.run.smithi086.stdout: bdev enable discard = True 2024-03-25T18:08:44.056 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore allocator = stupid 2024-03-25T18:08:44.056 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore block size = 96636764160 2024-03-25T18:08:44.056 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore fsck on mount = True 2024-03-25T18:08:44.056 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore zero block detection = True 2024-03-25T18:08:44.056 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluefs = 1 2024-03-25T18:08:44.057 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluestore = 1 2024-03-25T18:08:44.057 INFO:teuthology.orchestra.run.smithi086.stdout: debug monc = 20 2024-03-25T18:08:44.057 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-03-25T18:08:44.057 INFO:teuthology.orchestra.run.smithi086.stdout: debug osd = 20 2024-03-25T18:08:44.057 INFO:teuthology.orchestra.run.smithi086.stdout: debug rocksdb = 10 2024-03-25T18:08:44.057 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:08:44.057 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd full ratio = 0.9 2024-03-25T18:08:44.057 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:08:44.057 INFO:teuthology.orchestra.run.smithi086.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:08:44.057 INFO:teuthology.orchestra.run.smithi086.stdout: osd class default list = * 2024-03-25T18:08:44.057 INFO:teuthology.orchestra.run.smithi086.stdout: osd class load list = * 2024-03-25T18:08:44.057 INFO:teuthology.orchestra.run.smithi086.stdout: osd client watch timeout = 120 2024-03-25T18:08:44.057 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify cached snaps = True 2024-03-25T18:08:44.057 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify missing on start = True 2024-03-25T18:08:44.057 INFO:teuthology.orchestra.run.smithi086.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:08:44.058 INFO:teuthology.orchestra.run.smithi086.stdout: osd fast shutdown = False 2024-03-25T18:08:44.058 INFO:teuthology.orchestra.run.smithi086.stdout: osd max markdown count = 1000 2024-03-25T18:08:44.058 INFO:teuthology.orchestra.run.smithi086.stdout: osd mclock override recovery settings = True 2024-03-25T18:08:44.058 INFO:teuthology.orchestra.run.smithi086.stdout: osd objectstore = bluestore 2024-03-25T18:08:44.058 INFO:teuthology.orchestra.run.smithi086.stdout: osd op queue = debug_random 2024-03-25T18:08:44.058 INFO:teuthology.orchestra.run.smithi086.stdout: osd op queue cut off = debug_random 2024-03-25T18:08:44.058 INFO:teuthology.orchestra.run.smithi086.stdout: osd shutdown pgref assert = True 2024-03-25T18:08:44.058 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.058 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr] 2024-03-25T18:08:44.058 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-03-25T18:08:44.058 INFO:teuthology.orchestra.run.smithi086.stdout: debug mgr = 20 2024-03-25T18:08:44.058 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2024-03-25T18:08:44.058 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2024-03-25T18:08:44.058 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:08:44.059 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:08:44.059 INFO:teuthology.orchestra.run.smithi086.stdout: mgr/telemetry/nag = false 2024-03-25T18:08:44.059 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.059 INFO:teuthology.orchestra.run.smithi086.stdout:[mon] 2024-03-25T18:08:44.059 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-03-25T18:08:44.059 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2024-03-25T18:08:44.059 INFO:teuthology.orchestra.run.smithi086.stdout: debug paxos = 20 2024-03-25T18:08:44.059 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2024-03-25T18:08:44.059 INFO:teuthology.orchestra.run.smithi086.stdout: mon data avail warn = 5 2024-03-25T18:08:44.059 INFO:teuthology.orchestra.run.smithi086.stdout: mon mgr mkfs grace = 240 2024-03-25T18:08:44.059 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:08:44.059 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd reporter subtree level = osd 2024-03-25T18:08:44.059 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd prime pg temp = true 2024-03-25T18:08:44.059 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:08:44.059 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.060 INFO:teuthology.orchestra.run.smithi086.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:08:44.060 INFO:teuthology.orchestra.run.smithi086.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:08:44.060 INFO:teuthology.orchestra.run.smithi086.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:08:44.060 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.060 INFO:teuthology.orchestra.run.smithi086.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:08:44.060 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:08:44.060 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:08:44.060 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.060 INFO:teuthology.orchestra.run.smithi086.stdout: # 1m isn't quite enough 2024-03-25T18:08:44.060 INFO:teuthology.orchestra.run.smithi086.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:08:44.060 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.060 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:08:44.060 INFO:teuthology.orchestra.run.smithi086.stdout: mon min osdmap epochs = 50 2024-03-25T18:08:44.060 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd crush smoke test = False 2024-03-25T18:08:44.060 INFO:teuthology.orchestra.run.smithi086.stdout: mon osdmap full prune interval = 2 2024-03-25T18:08:44.060 INFO:teuthology.orchestra.run.smithi086.stdout: mon osdmap full prune min = 15 2024-03-25T18:08:44.061 INFO:teuthology.orchestra.run.smithi086.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:08:44.061 INFO:teuthology.orchestra.run.smithi086.stdout: mon scrub interval = 300 2024-03-25T18:08:44.061 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on pool no app = False 2024-03-25T18:08:44.061 INFO:teuthology.orchestra.run.smithi086.stdout: paxos service trim min = 10 2024-03-25T18:08:44.061 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-25T18:08:44.061 INFO:teuthology.orchestra.run.smithi086.stdout:[client] 2024-03-25T18:08:44.061 INFO:teuthology.orchestra.run.smithi086.stdout: rgw cache enabled = true 2024-03-25T18:08:44.061 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable ops log = true 2024-03-25T18:08:44.061 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable usage log = true 2024-03-25T18:08:44.061 INFO:teuthology.orchestra.run.smithi086.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:08:44.061 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:08:44.061 INFO:teuthology.orchestra.run.smithi086.stdout: debug monc = 20 2024-03-25T18:08:44.061 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-03-25T18:08:44.061 INFO:teuthology.orchestra.run.smithi086.stdout: debug objecter = 20 2024-03-25T18:08:44.061 INFO:teuthology.orchestra.run.smithi086.stdout: debug rados = 20 2024-03-25T18:08:44.062 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.a] 2024-03-25T18:08:44.062 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.c] 2024-03-25T18:08:44.062 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.b] 2024-03-25T18:08:44.063 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:08:44.063 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:08:44.117 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:08:44.117 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:08:44.118 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:08:44.132 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:08:44.132 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:08:44.186 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi064.front.sepia.ceph.com 2024-03-25T18:08:44.186 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:08:44.187 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:08:44.187 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:08:44.265 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:08:44.265 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:08:44.319 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi086.front.sepia.ceph.com 2024-03-25T18:08:44.320 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T18:08:44.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:08:44.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:08:44.359 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T18:08:44.359 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:08:44.414 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:08:44.414 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:08:44.414 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:08:44.468 INFO:teuthology.orchestra.run.smithi064.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:08:44.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:08:44.527 INFO:teuthology.orchestra.run.smithi086.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:08:44.530 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:08:44.530 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:08:44.530 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:08:44.572 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:08:44.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:08:44.629 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:08:44.644 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:08:44.644 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:08:44.644 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:08:44.670 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:08:44.671 INFO:tasks.ceph:role: osd.0 2024-03-25T18:08:44.671 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-25T18:08:44.671 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:08:44.744 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:08:44.744 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:08:44.744 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:08:44.744 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:08:44.744 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:08:44.744 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:08:44.744 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:08:44.744 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:08:44.744 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:08:44.745 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:08:45.247 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-03-25T18:08:45.249 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-03-25T18:08:45.249 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:08:45.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:08:45.426 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:08:45.493 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:08:45.493 INFO:tasks.ceph:role: osd.1 2024-03-25T18:08:45.493 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-25T18:08:45.494 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:08:45.567 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:08:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:08:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:08:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:08:45.568 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:08:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:08:45.568 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:08:45.568 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:08:45.568 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:08:45.568 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:08:46.026 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-03-25T18:08:46.028 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-03-25T18:08:46.029 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:08:46.115 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:08:46.147 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:08:46.215 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:08:46.215 INFO:tasks.ceph:role: osd.2 2024-03-25T18:08:46.215 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-25T18:08:46.215 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:08:46.289 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:08:46.289 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:08:46.289 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:08:46.290 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:08:46.290 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:08:46.290 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:08:46.290 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:08:46.290 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:08:46.290 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:08:46.290 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:08:46.772 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-03-25T18:08:46.774 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-03-25T18:08:46.774 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:08:46.928 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:08:46.964 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:08:47.033 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:08:47.033 INFO:tasks.ceph:role: osd.3 2024-03-25T18:08:47.033 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-25T18:08:47.033 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:08:47.109 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:08:47.109 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:08:47.109 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:08:47.109 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:08:47.109 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:08:47.110 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:08:47.110 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:08:47.110 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:08:47.110 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:08:47.110 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:08:47.634 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-03-25T18:08:47.636 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-03-25T18:08:47.636 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:08:47.789 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:08:47.826 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:08:47.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:47.929+0000 7faf6000d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:08:47.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:47.929+0000 7faf6000d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:08:47.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:47.929+0000 7faf6000d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:08:47.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:47.929+0000 7faf6000d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:08:47.930 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:47.929+0000 7faf6000d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:08:47.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:47.930+0000 7faf6000d640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:08:50.777 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:08:50.808 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:08:50.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:50.898+0000 7fa7a0868640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:08:50.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:50.898+0000 7fa7a0868640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:08:50.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:50.898+0000 7fa7a0868640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:08:50.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:50.898+0000 7fa7a0868640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:08:50.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:50.898+0000 7fa7a0868640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:08:50.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:50.899+0000 7fa7a0868640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:08:53.752 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:08:53.783 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:08:53.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:53.874+0000 7fc7e48d3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:08:53.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:53.874+0000 7fc7e48d3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:08:53.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:53.874+0000 7fc7e48d3640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:08:53.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:53.874+0000 7fc7e48d3640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:08:53.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:53.874+0000 7fc7e48d3640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:08:53.875 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:53.875+0000 7fc7e48d3640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:08:56.725 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:08:56.756 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:08:56.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:56.847+0000 7f11cab06640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:08:56.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:56.847+0000 7f11cab06640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:08:56.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:56.847+0000 7f11cab06640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:08:56.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:56.847+0000 7f11cab06640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:08:56.848 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:56.847+0000 7f11cab06640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:08:56.849 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:08:56.848+0000 7f11cab06640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:08:59.690 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:08:59.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:08:59.752 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:08:59.752 INFO:tasks.ceph:role: osd.4 2024-03-25T18:08:59.752 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-25T18:08:59.752 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:08:59.829 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:08:59.830 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:08:59.830 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:08:59.830 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:08:59.830 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:08:59.830 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:08:59.830 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:08:59.830 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:08:59.830 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:08:59.830 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:09:00.346 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-03-25T18:09:00.348 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-03-25T18:09:00.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:09:00.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:09:00.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:09:00.601 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:09:00.601 INFO:tasks.ceph:role: osd.5 2024-03-25T18:09:00.601 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-25T18:09:00.602 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:09:00.676 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:09:00.676 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:09:00.677 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:09:00.677 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:09:00.677 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:09:00.677 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:09:00.677 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:09:00.677 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:09:00.677 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:09:00.677 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:09:01.132 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-03-25T18:09:01.134 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-03-25T18:09:01.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:09:01.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:09:01.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:09:01.339 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:09:01.339 INFO:tasks.ceph:role: osd.6 2024-03-25T18:09:01.339 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-25T18:09:01.340 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:09:01.415 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:09:01.415 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:09:01.415 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:09:01.415 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:09:01.416 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:09:01.416 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:09:01.416 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:09:01.416 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:09:01.416 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:09:01.416 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:09:01.905 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-03-25T18:09:01.907 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-03-25T18:09:01.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:09:02.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:09:02.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:09:02.151 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:09:02.151 INFO:tasks.ceph:role: osd.7 2024-03-25T18:09:02.151 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-25T18:09:02.151 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:09:02.228 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:09:02.228 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:09:02.228 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:09:02.228 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:09:02.228 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:09:02.228 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:09:02.228 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:09:02.228 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:09:02.228 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:09:02.228 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:09:02.745 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-03-25T18:09:02.747 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-03-25T18:09:02.747 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:09:02.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:09:02.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:09:03.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:03.011+0000 7fdbc801e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:09:03.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:03.011+0000 7fdbc801e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:09:03.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:03.011+0000 7fdbc801e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:09:03.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:03.011+0000 7fdbc801e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:09:03.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:03.011+0000 7fdbc801e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:09:03.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:03.012+0000 7fdbc801e640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:09:05.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:09:05.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:09:05.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:05.971+0000 7f736b5e1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:09:05.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:05.972+0000 7f736b5e1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:09:05.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:05.972+0000 7f736b5e1640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:09:05.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:05.972+0000 7f736b5e1640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:09:05.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:05.972+0000 7f736b5e1640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:09:05.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:05.972+0000 7f736b5e1640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:09:08.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:09:08.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:09:08.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:08.940+0000 7ff4d5813640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:09:08.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:08.940+0000 7ff4d5813640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:09:08.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:08.940+0000 7ff4d5813640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:09:08.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:08.940+0000 7ff4d5813640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:09:08.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:08.940+0000 7ff4d5813640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:09:08.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:08.941+0000 7ff4d5813640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:09:11.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:09:11.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:09:11.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:11.907+0000 7f5d82811640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:09:11.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:11.907+0000 7f5d82811640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:09:11.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:11.907+0000 7f5d82811640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:09:11.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:11.907+0000 7f5d82811640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:09:11.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:11.907+0000 7f5d82811640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:09:11.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:11.908+0000 7f5d82811640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:09:14.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:09:14.780 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:09:14.780 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:09:14.780 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:09:14.812 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:09:14.812 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:09:14.883 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:09:14.883 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:09:14.953 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:09:14.953 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:09:15.023 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:09:15.023 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:09:15.092 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T18:09:15.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:09:15.124 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T18:09:15.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:09:15.194 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T18:09:15.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:09:15.263 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T18:09:15.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:09:15.332 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T18:09:15.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:09:15.401 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:09:15.401 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:09:15.419 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T18:09:15.419 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:09:15.456 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:09:15.456 DEBUG:teuthology.orchestra.run.smithi064:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:09:15.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:09:15.499 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.y] 2024-03-25T18:09:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCsvQFmgqLxGxAAskB2D4WDh+9yU3udys1DHg== 2024-03-25T18:09:15.499 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.0] 2024-03-25T18:09:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCvvQFml6BzNxAAMqaHrP3MSJ9ulzhHSe1/rQ== 2024-03-25T18:09:15.500 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.1] 2024-03-25T18:09:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCyvQFmVeGcNRAAvhwq6snSJik6KaV3M6Oz6g== 2024-03-25T18:09:15.500 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.2] 2024-03-25T18:09:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC1vQFmdx4oNBAA+c6KlNCbjX3hsFn1Q/vaDA== 2024-03-25T18:09:15.500 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.3] 2024-03-25T18:09:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC4vQFm7suRMhAA6LdaMDQ5k9LzWfKWk6D8cA== 2024-03-25T18:09:15.500 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.x] 2024-03-25T18:09:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCsvQFmNVZ/HxAA6teaWvaNMsaJnW6fDeXhNw== 2024-03-25T18:09:15.500 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.4] 2024-03-25T18:09:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC/vQFmGFu8ABAAyV/XErBLt7LqMZSLbTPNvw== 2024-03-25T18:09:15.500 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.5] 2024-03-25T18:09:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDBvQFmKIj5ORAAxKOrt87EG1Vui4/dautzlg== 2024-03-25T18:09:15.500 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.6] 2024-03-25T18:09:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDEvQFmifoaOBAAdJ7TKY2qJyC+CIh5PlUDGQ== 2024-03-25T18:09:15.500 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.7] 2024-03-25T18:09:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDHvQFmjf0oNhAA6q7folEg39eSHKdtuq8EZw== 2024-03-25T18:09:15.501 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2024-03-25T18:09:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCsvQFmpYYoIhAArNj6cmbG2GCjWVrw72/dIw== 2024-03-25T18:09:15.501 INFO:teuthology.orchestra.run.smithi064.stdout:[client.1] 2024-03-25T18:09:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCsvQFmY1WNJRAAiQWdq3nabVkl6ifgOeClZQ== 2024-03-25T18:09:15.522 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.y] 2024-03-25T18:09:15.522 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCsvQFmgqLxGxAAskB2D4WDh+9yU3udys1DHg== 2024-03-25T18:09:15.522 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.0] 2024-03-25T18:09:15.522 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCvvQFml6BzNxAAMqaHrP3MSJ9ulzhHSe1/rQ== 2024-03-25T18:09:15.522 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.1] 2024-03-25T18:09:15.523 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCyvQFmVeGcNRAAvhwq6snSJik6KaV3M6Oz6g== 2024-03-25T18:09:15.523 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.2] 2024-03-25T18:09:15.523 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQC1vQFmdx4oNBAA+c6KlNCbjX3hsFn1Q/vaDA== 2024-03-25T18:09:15.523 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.3] 2024-03-25T18:09:15.523 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQC4vQFm7suRMhAA6LdaMDQ5k9LzWfKWk6D8cA== 2024-03-25T18:09:15.523 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.x] 2024-03-25T18:09:15.523 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCsvQFmNVZ/HxAA6teaWvaNMsaJnW6fDeXhNw== 2024-03-25T18:09:15.523 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.4] 2024-03-25T18:09:15.523 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQC/vQFmGFu8ABAAyV/XErBLt7LqMZSLbTPNvw== 2024-03-25T18:09:15.523 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.5] 2024-03-25T18:09:15.523 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDBvQFmKIj5ORAAxKOrt87EG1Vui4/dautzlg== 2024-03-25T18:09:15.523 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.6] 2024-03-25T18:09:15.523 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDEvQFmifoaOBAAdJ7TKY2qJyC+CIh5PlUDGQ== 2024-03-25T18:09:15.523 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.7] 2024-03-25T18:09:15.523 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDHvQFmjf0oNhAA6q7folEg39eSHKdtuq8EZw== 2024-03-25T18:09:15.523 INFO:teuthology.orchestra.run.smithi086.stdout:[client.0] 2024-03-25T18:09:15.524 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCsvQFmpYYoIhAArNj6cmbG2GCjWVrw72/dIw== 2024-03-25T18:09:15.524 INFO:teuthology.orchestra.run.smithi086.stdout:[client.1] 2024-03-25T18:09:15.524 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCsvQFmY1WNJRAAiQWdq3nabVkl6ifgOeClZQ== 2024-03-25T18:09:15.524 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:09:15.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:09:15.617 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:09:15.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:09:15.670 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:09:15.672 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:09:15.724 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:09:15.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:09:15.777 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:09:15.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:09:15.831 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:09:15.833 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:09:15.884 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:09:15.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:09:15.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:09:15.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:09:15.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:09:15.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:09:16.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:09:16.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:09:16.096 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:09:16.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:09:16.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:09:16.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:09:16.210 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:09:16.210 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:09:16.236 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:09:16.332 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:09:16.358 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:09:16.425 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:09:16.523 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:09:16.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:09:16.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:09:16.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:09:16.700 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:09:16.702 DEBUG:teuthology.orchestra.run.smithi086:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:09:16.755 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:09:16.755 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:09:16.755 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:09:16.755 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:09:16.758 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:09:16.758 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:09:16.758 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:09:16.758 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:09:16.760 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:09:16.760 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:09:16.760 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:09:16.760 DEBUG:teuthology.orchestra.run.smithi086:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:09:16.798 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:09:16.798 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:09:16.798 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:09:16.798 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:09:16.800 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:09:16.801 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:09:16.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:09:16.803 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:09:16.803 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:09:16.803 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:09:16.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.903+0000 7f6071bb5640 10 monclient: get_monmap_and_config 2024-03-25T18:09:16.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.903+0000 7f6071bb5640 10 monclient: build_initial_monmap 2024-03-25T18:09:16.904 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.903+0000 7f6071bb5640 10 monclient: monmap: 2024-03-25T18:09:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-03-25T18:09:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-25T18:09:16.904775+0000 2024-03-25T18:09:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-25T18:09:16.904775+0000 2024-03-25T18:09:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:09:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:09:16.905 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-25T18:09:16.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.904+0000 7f6071bb5640 1 Processor -- start 2024-03-25T18:09:16.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.904+0000 7f6071bb5640 1 -- start start 2024-03-25T18:09:16.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.904+0000 7f6071bb5640 10 monclient: init 2024-03-25T18:09:16.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.905+0000 7f6071bb5640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:16.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.905+0000 7f6071bb5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:16.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.905+0000 7f6071bb5640 10 monclient(hunting): picked mon.noname-c con 0x7f606c0b4700 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:16.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.905+0000 7f6071bb5640 10 monclient(hunting): picked mon.noname-b con 0x7f606c142a20 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:16.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.905+0000 7f6071bb5640 10 monclient(hunting): picked mon.noname-a con 0x7f606c0a96b0 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:16.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.905+0000 7f6071bb5640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c0a9d60 con 0x7f606c0a96b0 2024-03-25T18:09:16.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.905+0000 7f6071bb5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c13ff10 con 0x7f606c0b4700 2024-03-25T18:09:16.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.905+0000 7f6071bb5640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c0aa1f0 con 0x7f606c142a20 2024-03-25T18:09:16.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.905+0000 7f6071bb5640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:16.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.905+0000 7f6071bb5640 10 monclient(hunting): authenticate will time out at 6943.143807s 2024-03-25T18:09:16.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.905+0000 7f606bfff640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f606c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-03-25T18:09:16.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.905+0000 7f606affd640 1 -- >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f606c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6790/0 2024-03-25T18:09:16.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:16.905+0000 7f606b7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f606c0b4700 legacy=0x7f606c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-03-25T18:09:17.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:17.106+0000 7f606bfff640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f606c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-03-25T18:09:17.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:17.106+0000 7f606affd640 1 -- >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f606c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6790/0 2024-03-25T18:09:17.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:17.106+0000 7f606b7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f606c0b4700 legacy=0x7f606c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-03-25T18:09:17.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:17.507+0000 7f606affd640 1 -- >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f606c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6790/0 2024-03-25T18:09:17.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:17.507+0000 7f606bfff640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f606c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-03-25T18:09:17.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:17.507+0000 7f606b7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f606c0b4700 legacy=0x7f606c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-03-25T18:09:18.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:18.308+0000 7f606bfff640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f606c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-03-25T18:09:18.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:18.308+0000 7f606affd640 1 -- >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f606c13f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6790/0 2024-03-25T18:09:18.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:18.308+0000 7f606b7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f606c0b4700 legacy=0x7f606c142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-03-25T18:09:19.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:19.905+0000 7f60697fa640 10 monclient(hunting): tick 2024-03-25T18:09:19.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:19.905+0000 7f60697fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:09:19.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:19.905+0000 7f60697fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:09:19.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:19.905+0000 7f60697fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:09:19.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:19.905+0000 7f60697fa640 1 -- >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f606c13f0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:09:19.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:19.905+0000 7f60697fa640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f606c0b4700 legacy=0x7f606c142310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:09:19.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:19.905+0000 7f60697fa640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f606c13f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:09:19.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:19.905+0000 7f60697fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:19.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:19.905+0000 7f60697fa640 10 monclient(hunting): picked mon.noname-c con 0x7f6058003680 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:19.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:19.905+0000 7f60697fa640 10 monclient(hunting): picked mon.noname-b con 0x7f606c142a20 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:19.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:19.905+0000 7f60697fa640 10 monclient(hunting): picked mon.noname-a con 0x7f606c0a96b0 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:19.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:19.905+0000 7f60697fa640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c0b4ae0 con 0x7f606c0a96b0 2024-03-25T18:09:19.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:19.905+0000 7f60697fa640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c0a9d60 con 0x7f6058003680 2024-03-25T18:09:19.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:19.905+0000 7f60697fa640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c13ff10 con 0x7f606c142a20 2024-03-25T18:09:19.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:19.905+0000 7f60697fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:19.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:19.905+0000 7f606bfff640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f605800cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-03-25T18:09:19.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:19.905+0000 7f606affd640 1 -- >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f6058009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6790/0 2024-03-25T18:09:19.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:19.905+0000 7f606b7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f6058003680 legacy=0x7f6058005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-03-25T18:09:20.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:20.106+0000 7f606bfff640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f605800cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-03-25T18:09:20.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:20.106+0000 7f606affd640 1 -- >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f6058009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6790/0 2024-03-25T18:09:20.107 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:20.106+0000 7f606b7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f6058003680 legacy=0x7f6058005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-03-25T18:09:20.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:20.506+0000 7f606bfff640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f605800cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-03-25T18:09:20.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:20.506+0000 7f606affd640 1 -- >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f6058009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6790/0 2024-03-25T18:09:20.508 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:20.506+0000 7f606b7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f6058003680 legacy=0x7f6058005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-03-25T18:09:21.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:21.307+0000 7f606bfff640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f605800cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-03-25T18:09:21.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:21.307+0000 7f606affd640 1 -- >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f6058009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6790/0 2024-03-25T18:09:21.309 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:21.307+0000 7f606b7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f6058003680 legacy=0x7f6058005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-03-25T18:09:22.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:22.904+0000 7f60697fa640 10 monclient(hunting): tick 2024-03-25T18:09:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:22.904+0000 7f60697fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:09:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:22.904+0000 7f60697fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:09:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:22.904+0000 7f60697fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:09:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:22.904+0000 7f60697fa640 1 -- >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f6058009300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:09:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:22.904+0000 7f60697fa640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f6058003680 legacy=0x7f6058005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:09:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:22.904+0000 7f60697fa640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f605800cbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:09:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:22.904+0000 7f60697fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:09:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:22.904+0000 7f60697fa640 10 monclient(hunting): picked mon.noname-c con 0x7f6058003680 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:22.904+0000 7f60697fa640 10 monclient(hunting): picked mon.noname-a con 0x7f606c0a96b0 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:22.904+0000 7f60697fa640 10 monclient(hunting): picked mon.noname-b con 0x7f606c142a20 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:22.904+0000 7f60697fa640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c0aa1f0 con 0x7f606c0a96b0 2024-03-25T18:09:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:22.904+0000 7f60697fa640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c0b4ae0 con 0x7f6058003680 2024-03-25T18:09:22.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:22.904+0000 7f60697fa640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c0a9d60 con 0x7f606c142a20 2024-03-25T18:09:22.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:22.904+0000 7f60697fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:22.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:22.904+0000 7f606affd640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f6058005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-03-25T18:09:22.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:22.904+0000 7f606bfff640 1 -- >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f6058009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6790/0 2024-03-25T18:09:22.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:22.904+0000 7f606b7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f6058003680 legacy=0x7f6058010430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-03-25T18:09:23.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:23.105+0000 7f606bfff640 1 -- >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f6058009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6790/0 2024-03-25T18:09:23.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:23.105+0000 7f606affd640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f6058005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-03-25T18:09:23.106 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:23.105+0000 7f606b7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f6058003680 legacy=0x7f6058010430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-03-25T18:09:23.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:23.506+0000 7f606affd640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f6058005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-03-25T18:09:23.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:23.506+0000 7f606bfff640 1 -- >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f6058009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6790/0 2024-03-25T18:09:23.507 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:23.506+0000 7f606b7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f6058003680 legacy=0x7f6058010430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-03-25T18:09:23.780 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:00:06.859 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:24.085 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:00:07.164 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:24.276 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:00:07.389 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:24.278 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:00:07.391 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:24.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:24.307+0000 7f606affd640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f6058005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6789/0 2024-03-25T18:09:24.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:24.307+0000 7f606bfff640 1 -- >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f6058009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.64:6790/0 2024-03-25T18:09:24.308 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:24.307+0000 7f606b7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f6058003680 legacy=0x7f6058010430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-03-25T18:09:24.604 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:00:07.716 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:24.604 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:00:07.717 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:25.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:25.904+0000 7f60697fa640 10 monclient(hunting): tick 2024-03-25T18:09:25.906 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:25.904+0000 7f60697fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:09:25.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:25.904+0000 7f60697fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:09:25.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:25.904+0000 7f60697fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:09:25.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:25.904+0000 7f60697fa640 1 -- >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f6058009300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:09:25.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:25.904+0000 7f60697fa640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f6058003680 legacy=0x7f6058010430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:09:25.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:25.904+0000 7f60697fa640 1 -- >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f6058005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:09:25.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:25.904+0000 7f60697fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:09:25.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:25.904+0000 7f60697fa640 10 monclient(hunting): picked mon.noname-c con 0x7f6058003680 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:25.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:25.904+0000 7f60697fa640 10 monclient(hunting): picked mon.noname-a con 0x7f606c0a96b0 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:25.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:25.904+0000 7f60697fa640 10 monclient(hunting): picked mon.noname-b con 0x7f606c142a20 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:25.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:25.904+0000 7f60697fa640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c13ff10 con 0x7f606c0a96b0 2024-03-25T18:09:25.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:25.904+0000 7f60697fa640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c0aa1f0 con 0x7f6058003680 2024-03-25T18:09:25.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:25.904+0000 7f60697fa640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c0b4ae0 con 0x7f606c142a20 2024-03-25T18:09:25.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:25.904+0000 7f60697fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:25.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:25.961+0000 7f606b7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6058003680 0x7f605800cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:40640/0 (socket says 172.21.15.64:40640) 2024-03-25T18:09:25.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:25.961+0000 7f606b7fe640 1 -- 172.21.15.64:0/4150847920 learned_addr learned my addr 172.21.15.64:0/4150847920 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:09:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:26.429+0000 7f606bfff640 1 -- 172.21.15.64:0/4150847920 >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f6058009300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T18:09:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:26.429+0000 7f606bfff640 1 -- 172.21.15.64:0/4150847920 >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f6058009300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:09:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:26.429+0000 7f606bfff640 1 --1- 172.21.15.64:0/4150847920 >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 0x7f6058009300 :-1 s=OPENED pgs=2 cs=1 l=1).handle_message read tag failed 2024-03-25T18:09:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:26.429+0000 7f606bfff640 1 --1- 172.21.15.64:0/4150847920 >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 0x7f6058009300 :-1 s=OPENED pgs=2 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:09:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:26.429+0000 7f606a7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.64:6790/0 2024-03-25T18:09:26.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:26.446+0000 7f606affd640 1 -- 172.21.15.64:0/4150847920 >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f6058010430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T18:09:26.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:26.446+0000 7f606affd640 1 -- 172.21.15.64:0/4150847920 >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f6058010430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:09:26.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:26.446+0000 7f606affd640 1 --1- 172.21.15.64:0/4150847920 >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 0x7f6058010430 :-1 s=OPENED pgs=2 cs=1 l=1).handle_message read tag failed 2024-03-25T18:09:26.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:26.446+0000 7f606affd640 1 --1- 172.21.15.64:0/4150847920 >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 0x7f6058010430 :-1 s=OPENED pgs=2 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:09:26.448 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:26.446+0000 7f606a7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.64:6789/0 2024-03-25T18:09:28.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:28.906+0000 7f60697fa640 10 monclient(hunting): tick 2024-03-25T18:09:28.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:28.906+0000 7f60697fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:09:28.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:28.906+0000 7f60697fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:09:28.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:28.906+0000 7f60697fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:09:28.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:28.906+0000 7f60697fa640 1 -- 172.21.15.64:0/4150847920 >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f6058009300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:09:28.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:28.906+0000 7f60697fa640 1 -- 172.21.15.64:0/4150847920 >> v1:172.21.15.86:6789/0 conn(0x7f6058003680 legacy=0x7f605800cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:28.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:28.906+0000 7f60697fa640 1 -- 172.21.15.64:0/4150847920 >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f6058010430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:09:28.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:28.906+0000 7f60697fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:09:28.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:28.906+0000 7f60697fa640 10 monclient(hunting): picked mon.noname-b con 0x7f606c142a20 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:28.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:28.906+0000 7f60697fa640 10 monclient(hunting): picked mon.noname-a con 0x7f606c0a96b0 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:28.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:28.906+0000 7f60697fa640 10 monclient(hunting): picked mon.noname-c con 0x7f6058003680 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:28.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:28.906+0000 7f60697fa640 1 -- 172.21.15.64:0/4150847920 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c0a9d60 con 0x7f606c0a96b0 2024-03-25T18:09:28.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:28.906+0000 7f60697fa640 1 -- 172.21.15.64:0/4150847920 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6058006b30 con 0x7f6058003680 2024-03-25T18:09:28.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:28.906+0000 7f60697fa640 1 -- 172.21.15.64:0/4150847920 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6058006d60 con 0x7f606c142a20 2024-03-25T18:09:28.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:28.906+0000 7f60697fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:31.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:31.906+0000 7f60697fa640 10 monclient(hunting): tick 2024-03-25T18:09:31.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:31.906+0000 7f60697fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:09:31.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:31.906+0000 7f60697fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:09:31.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:31.906+0000 7f60697fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:09:31.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:31.906+0000 7f60697fa640 1 -- 172.21.15.64:0/4150847920 >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f6058010c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:31.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:31.906+0000 7f60697fa640 1 -- 172.21.15.64:0/4150847920 >> v1:172.21.15.86:6789/0 conn(0x7f6058003680 legacy=0x7f605800cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:31.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:31.906+0000 7f60697fa640 1 -- 172.21.15.64:0/4150847920 >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f6058010430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:31.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:31.906+0000 7f60697fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:31.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:31.906+0000 7f60697fa640 10 monclient(hunting): picked mon.noname-c con 0x7f60580072f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:31.907 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:31.906+0000 7f60697fa640 10 monclient(hunting): picked mon.noname-b con 0x7f606c142a20 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:31.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:31.906+0000 7f60697fa640 10 monclient(hunting): picked mon.noname-a con 0x7f606c0a96b0 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:31.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:31.906+0000 7f60697fa640 1 -- 172.21.15.64:0/4150847920 --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6058006fb0 con 0x7f606c0a96b0 2024-03-25T18:09:31.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:31.906+0000 7f60697fa640 1 -- 172.21.15.64:0/4150847920 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6058009810 con 0x7f60580072f0 2024-03-25T18:09:31.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:31.906+0000 7f60697fa640 1 -- 172.21.15.64:0/4150847920 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6058010c70 con 0x7f606c142a20 2024-03-25T18:09:31.908 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:31.906+0000 7f60697fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:34.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.734+0000 7f606a7fc640 1 -- 172.21.15.64:0/4150847920 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130334005 0 0) 0x7f6058010c70 con 0x7f606c142a20 2024-03-25T18:09:34.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.734+0000 7f606a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:34.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.734+0000 7f606a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:34.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.734+0000 7f606a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:34.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.734+0000 7f606a7fc640 10 monclient(hunting): my global_id is 4103 2024-03-25T18:09:34.735 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.734+0000 7f606a7fc640 1 -- 172.21.15.64:0/4150847920 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6054003520 con 0x7f606c142a20 2024-03-25T18:09:34.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.765+0000 7f606a7fc640 1 -- 172.21.15.64:0/4150847920 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070055181 0 0) 0x7f6058006fb0 con 0x7f606c0a96b0 2024-03-25T18:09:34.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.765+0000 7f606a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:34.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.765+0000 7f606a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:34.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.765+0000 7f606a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:34.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.765+0000 7f606a7fc640 10 monclient(hunting): my global_id is 4104 2024-03-25T18:09:34.766 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.766+0000 7f606a7fc640 1 -- 172.21.15.64:0/4150847920 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6058010c70 con 0x7f606c0a96b0 2024-03-25T18:09:34.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.770+0000 7f606a7fc640 1 -- 172.21.15.64:0/4150847920 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447109091 0 0) 0x7f6058009810 con 0x7f60580072f0 2024-03-25T18:09:34.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.770+0000 7f606a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:34.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.770+0000 7f606a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:34.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.770+0000 7f606a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:34.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.770+0000 7f606a7fc640 10 monclient(hunting): my global_id is 4105 2024-03-25T18:09:34.770 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.770+0000 7f606a7fc640 1 -- 172.21.15.64:0/4150847920 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6058006fb0 con 0x7f60580072f0 2024-03-25T18:09:34.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.795+0000 7f606a7fc640 1 -- 172.21.15.64:0/4150847920 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 544499627 0 0) 0x7f6054003520 con 0x7f606c142a20 2024-03-25T18:09:34.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.795+0000 7f606a7fc640 1 -- 172.21.15.64:0/4150847920 >> v1:172.21.15.86:6789/0 conn(0x7f60580072f0 legacy=0x7f6058011510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:34.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.796+0000 7f606a7fc640 1 -- 172.21.15.64:0/4150847920 >> v1:172.21.15.64:6789/0 conn(0x7f606c0a96b0 legacy=0x7f6058010430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:34.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.796+0000 7f606a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:09:34.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.796+0000 7f606a7fc640 1 monclient: found mon.noname-b 2024-03-25T18:09:34.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.796+0000 7f606a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:34.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.796+0000 7f606a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:09:34.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.796+0000 7f606a7fc640 1 -- 172.21.15.64:0/4150847920 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6058010ea0 con 0x7f606c142a20 2024-03-25T18:09:34.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.796+0000 7f606a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:09:34.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.796+0000 7f606a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:09:34.797 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.796+0000 7f606a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:34.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.796+0000 7f606a7fc640 1 -- 172.21.15.64:0/4150847920 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f6060004330 con 0x7f606c142a20 2024-03-25T18:09:34.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.796+0000 7f606a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:34.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.796+0000 7f606a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:34.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.796+0000 7f606a7fc640 10 monclient: dump: 2024-03-25T18:09:34.798 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-25T18:09:34.798 INFO:teuthology.orchestra.run.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:34.798 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:34.798 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:34.798 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:34.798 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:34.798 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:34.798 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:34.798 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:34.798 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-25T18:09:34.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.796+0000 7f6071bb5640 5 monclient: authenticate success, global_id 4103 2024-03-25T18:09:34.798 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.796+0000 7f6071bb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:34.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.896+0000 7f606a7fc640 1 -- 172.21.15.64:0/4150847920 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6060004490 con 0x7f606c142a20 2024-03-25T18:09:34.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.896+0000 7f606a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:34.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.896+0000 7f606a7fc640 1 -- 172.21.15.64:0/4150847920 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f60600048e0 con 0x7f606c142a20 2024-03-25T18:09:34.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.896+0000 7f606a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:34.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.896+0000 7f606a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:34.897 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.896+0000 7f606a7fc640 10 monclient: dump: 2024-03-25T18:09:34.897 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-25T18:09:34.897 INFO:teuthology.orchestra.run.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:34.897 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:34.897 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:34.897 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:34.897 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:34.897 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:34.898 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:34.898 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:34.898 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-25T18:09:34.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.896+0000 7f6071bb5640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:34.898 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.896+0000 7f6071bb5640 4 set_mon_vals no callback set 2024-03-25T18:09:34.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.897+0000 7f6071bb5640 10 monclient: shutdown 2024-03-25T18:09:34.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.897+0000 7f6071bb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:34.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.897+0000 7f6071bb5640 1 -- 172.21.15.64:0/4150847920 >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f605800cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:34.900 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.897+0000 7f6071bb5640 1 -- 172.21.15.64:0/4150847920 shutdown_connections 2024-03-25T18:09:34.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.897+0000 7f6071bb5640 1 -- 172.21.15.64:0/4150847920 >> 172.21.15.64:0/4150847920 conn(0x7f606c0668e0 msgr2=0x7f606c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:34.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.897+0000 7f6071bb5640 1 -- 172.21.15.64:0/4150847920 shutdown_connections 2024-03-25T18:09:34.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.897+0000 7f6071bb5640 1 -- 172.21.15.64:0/4150847920 wait complete. 2024-03-25T18:09:34.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.897+0000 7f6071bb5640 10 monclient: build_initial_monmap 2024-03-25T18:09:34.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.897+0000 7f6071bb5640 10 monclient: monmap: 2024-03-25T18:09:34.901 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-03-25T18:09:34.901 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:34.901 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-03-25T18:09:34.901 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-03-25T18:09:34.901 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:34.901 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-25T18:09:34.901 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:34.901 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:09:34.901 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:09:34.901 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-25T18:09:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.898+0000 7f6071bb5640 1 librados: starting msgr at 2024-03-25T18:09:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.898+0000 7f6071bb5640 1 librados: starting objecter 2024-03-25T18:09:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.898+0000 7f6071bb5640 1 Processor -- start 2024-03-25T18:09:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.898+0000 7f6071bb5640 1 -- start start 2024-03-25T18:09:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.898+0000 7f6071bb5640 1 librados: setting wanted keys 2024-03-25T18:09:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.898+0000 7f6071bb5640 1 librados: calling monclient init 2024-03-25T18:09:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.898+0000 7f6071bb5640 10 monclient: init 2024-03-25T18:09:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.899+0000 7f6071bb5640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.899+0000 7f6071bb5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:09:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.899+0000 7f6071bb5640 10 monclient(hunting): picked mon.noname-a con 0x7f60580072f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.899+0000 7f6071bb5640 10 monclient(hunting): picked mon.noname-b con 0x7f606c0a96b0 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.899+0000 7f6071bb5640 10 monclient(hunting): picked mon.noname-c con 0x7f606c142a20 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.899+0000 7f6071bb5640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c0add30 con 0x7f60580072f0 2024-03-25T18:09:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.899+0000 7f6071bb5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c1c8800 con 0x7f606c0a96b0 2024-03-25T18:09:34.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.899+0000 7f6071bb5640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f606c1c95a0 con 0x7f606c142a20 2024-03-25T18:09:34.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.899+0000 7f6071bb5640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:34.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.899+0000 7f6071bb5640 10 monclient(hunting): authenticate will time out at 6961.137553s 2024-03-25T18:09:34.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.899+0000 7f606affd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f606c0a96b0 0x7f606c1d7ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:49974/0 (socket says 172.21.15.64:49974) 2024-03-25T18:09:34.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.899+0000 7f606affd640 1 -- 172.21.15.64:0/4046929405 learned_addr learned my addr 172.21.15.64:0/4046929405 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:09:34.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.900+0000 7f6053fff640 10 client.?.objecter ms_handle_connect 0x7f606c0a96b0 2024-03-25T18:09:34.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.900+0000 7f6053fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:34.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.900+0000 7f6053fff640 10 client.?.objecter ms_handle_connect 0x7f60580072f0 2024-03-25T18:09:34.903 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.900+0000 7f6053fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:34.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.937+0000 7f6053fff640 10 client.?.objecter ms_handle_connect 0x7f606c142a20 2024-03-25T18:09:34.938 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.937+0000 7f6053fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:34.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.954+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136766239 0 0) 0x7f606c1c8800 con 0x7f606c0a96b0 2024-03-25T18:09:34.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.954+0000 7f6053fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:34.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.954+0000 7f6053fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:34.954 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.954+0000 7f6053fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:34.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.954+0000 7f6053fff640 10 monclient(hunting): my global_id is 4114 2024-03-25T18:09:34.955 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.954+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6058009810 con 0x7f606c0a96b0 2024-03-25T18:09:34.961 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.961+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784070420 0 0) 0x7f606c1c95a0 con 0x7f606c142a20 2024-03-25T18:09:34.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.961+0000 7f6053fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:34.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.961+0000 7f6053fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:34.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.961+0000 7f6053fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:34.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.961+0000 7f6053fff640 10 monclient(hunting): my global_id is 4112 2024-03-25T18:09:34.962 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.961+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f606c1c8800 con 0x7f606c142a20 2024-03-25T18:09:34.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.969+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256895503 0 0) 0x7f606c0add30 con 0x7f60580072f0 2024-03-25T18:09:34.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.969+0000 7f6053fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:34.969 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.969+0000 7f6053fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:34.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.969+0000 7f6053fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:34.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.969+0000 7f6053fff640 10 monclient(hunting): my global_id is 4110 2024-03-25T18:09:34.970 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:34.969+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f606c1c95a0 con 0x7f60580072f0 2024-03-25T18:09:35.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.021+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3473720246 0 0) 0x7f6058009810 con 0x7f606c0a96b0 2024-03-25T18:09:35.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.021+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f606c0add30 con 0x7f606c0a96b0 2024-03-25T18:09:35.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.022+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f60600048e0 con 0x7f606c0a96b0 2024-03-25T18:09:35.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.022+0000 7f6053fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:35.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.022+0000 7f6053fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:35.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.022+0000 7f6053fff640 10 monclient(hunting): dump: 2024-03-25T18:09:35.023 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-25T18:09:35.023 INFO:teuthology.orchestra.run.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:35.023 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:35.023 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:35.023 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:35.023 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:35.023 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:35.023 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:35.023 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:35.023 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-25T18:09:35.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.022+0000 7f6053fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:35.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.024+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2143771613 0 0) 0x7f606c1c8800 con 0x7f606c142a20 2024-03-25T18:09:35.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.024+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6058009810 con 0x7f606c142a20 2024-03-25T18:09:35.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.024+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f60640040c0 con 0x7f606c142a20 2024-03-25T18:09:35.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.024+0000 7f6053fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:35.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.024+0000 7f6053fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:35.025 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.024+0000 7f6053fff640 10 monclient(hunting): dump: 2024-03-25T18:09:35.025 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-25T18:09:35.025 INFO:teuthology.orchestra.run.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:35.025 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:35.025 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:35.025 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:35.025 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:35.025 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:35.025 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:35.026 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:35.026 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-25T18:09:35.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2849098958 0 0) 0x7f6058009810 con 0x7f606c142a20 2024-03-25T18:09:35.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 >> v1:172.21.15.86:6789/0 conn(0x7f606c0a96b0 legacy=0x7f606c1d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:35.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 >> v1:172.21.15.64:6789/0 conn(0x7f60580072f0 legacy=0x7f606c13a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:35.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6053fff640 10 monclient: _finish_hunting 0 2024-03-25T18:09:35.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6053fff640 1 monclient: found mon.c 2024-03-25T18:09:35.034 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6053fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:35.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6053fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:35.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606c1c8420 con 0x7f606c142a20 2024-03-25T18:09:35.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6053fff640 10 monclient: _finish_auth 0 2024-03-25T18:09:35.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6053fff640 10 monclient: _check_auth_tickets 2024-03-25T18:09:35.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6053fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:35.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6071bb5640 5 monclient: authenticate success, global_id 4112 2024-03-25T18:09:35.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6071bb5640 10 monclient: _renew_subs 2024-03-25T18:09:35.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6071bb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:35.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6071bb5640 1 -- 172.21.15.64:0/4046929405 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f606c0b0670 con 0x7f606c142a20 2024-03-25T18:09:35.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6071bb5640 10 client.4112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:35.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6071bb5640 10 monclient: _renew_subs 2024-03-25T18:09:35.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6071bb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:35.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6071bb5640 1 -- 172.21.15.64:0/4046929405 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f606c0b1380 con 0x7f606c142a20 2024-03-25T18:09:35.035 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.033+0000 7f6071bb5640 1 librados: init done 2024-03-25T18:09:35.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.034+0000 7f6071bb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:35.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.034+0000 7f6071bb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:35.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.034+0000 7f6071bb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:35.036 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.034+0000 7f6071bb5640 1 -- 172.21.15.64:0/4046929405 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606c1c8610 con 0x7f606c142a20 2024-03-25T18:09:35.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.250+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60640044d0 con 0x7f606c142a20 2024-03-25T18:09:35.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.250+0000 7f6053fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:35.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.250+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f60640048e0 con 0x7f606c142a20 2024-03-25T18:09:35.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.250+0000 7f6053fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:35.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.250+0000 7f6053fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:35.251 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.250+0000 7f6053fff640 10 monclient: dump: 2024-03-25T18:09:35.251 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-25T18:09:35.251 INFO:teuthology.orchestra.run.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:35.251 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:35.251 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:35.251 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:35.251 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:35.251 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:35.251 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:35.252 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:35.252 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-25T18:09:35.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.250+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6064004e90 con 0x7f606c142a20 2024-03-25T18:09:35.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.250+0000 7f60697fa640 4 set_mon_vals no callback set 2024-03-25T18:09:35.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.250+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1415251544 0 0) 0x7f6064006340 con 0x7f606c142a20 2024-03-25T18:09:35.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.250+0000 7f6053fff640 10 client.4112.objecter ms_dispatch 0x7f606c000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:09:35.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.250+0000 7f6053fff640 3 client.4112.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:09:35.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.250+0000 7f6053fff640 3 client.4112.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:09:35.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.250+0000 7f6053fff640 20 client.4112.objecter dump_active .. 0 homeless 2024-03-25T18:09:35.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.250+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6064006780 con 0x7f606c142a20 2024-03-25T18:09:35.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.250+0000 7f6053fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:35.252 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.250+0000 7f6053fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:35.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.321+0000 7f6071bb5640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:09:35.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.321+0000 7f6071bb5640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:09:35.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.321+0000 7f6071bb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:35.322 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.321+0000 7f6071bb5640 1 -- 172.21.15.64:0/4046929405 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f606c04f730 con 0x7f606c142a20 2024-03-25T18:09:35.470 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:35.469+0000 7fb779810040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:09:35.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.477+0000 7f6053fff640 1 -- 172.21.15.64:0/4046929405 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f6064005400 con 0x7f606c142a20 2024-03-25T18:09:35.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.477+0000 7f6053fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:09:35.477 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.477+0000 7f6053fff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:09:35.477 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2024-03-25T18:09:35.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.477+0000 7f6071bb5640 10 librados: watch_flush enter 2024-03-25T18:09:35.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.477+0000 7f6071bb5640 10 librados: watch_flush exit 2024-03-25T18:09:35.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.477+0000 7f6071bb5640 20 client.4112.objecter shutdown clearing up homeless session... 2024-03-25T18:09:35.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.477+0000 7f6071bb5640 10 client.4112.objecter successfully canceled tick 2024-03-25T18:09:35.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.478+0000 7f6071bb5640 10 monclient: shutdown 2024-03-25T18:09:35.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.478+0000 7f6071bb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:35.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.478+0000 7f6071bb5640 1 -- 172.21.15.64:0/4046929405 >> v1:172.21.15.64:6790/0 conn(0x7f606c142a20 legacy=0x7f606c1e3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:35.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.478+0000 7f6071bb5640 1 -- 172.21.15.64:0/4046929405 shutdown_connections 2024-03-25T18:09:35.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.478+0000 7f6071bb5640 1 -- 172.21.15.64:0/4046929405 >> 172.21.15.64:0/4046929405 conn(0x7f606c0668e0 msgr2=0x7f606c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:35.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.478+0000 7f6071bb5640 1 -- 172.21.15.64:0/4046929405 shutdown_connections 2024-03-25T18:09:35.479 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.478+0000 7f6071bb5640 1 -- 172.21.15.64:0/4046929405 wait complete. 2024-03-25T18:09:35.480 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:35.478+0000 7f6071bb5640 1 librados: shutdown 2024-03-25T18:09:35.490 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:09:35.490 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:09:35.491 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:09:35.491 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:09:35.518 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:09:35.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:09:35.534 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:35.533+0000 7fb779810040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:09:35.559 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:09:35.560 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:09:35.588 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:35.587+0000 7f7080038040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:09:35.626 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-25T18:09:35.626 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:09:35.646 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:35.646+0000 7f7080038040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:09:35.694 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T18:09:35.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:09:35.721 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T18:09:35.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:09:35.788 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T18:09:35.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:09:35.855 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-25T18:09:35.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:09:35.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new b7d57e83-d493-43bf-8e30-09bcea0e9df8 0 2024-03-25T18:09:36.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.061+0000 7f118f772640 10 monclient: get_monmap_and_config 2024-03-25T18:09:36.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.061+0000 7f118f772640 10 monclient: build_initial_monmap 2024-03-25T18:09:36.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.061+0000 7f118f772640 10 monclient: monmap: 2024-03-25T18:09:36.063 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-25T18:09:36.063 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:36.063 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:09:36.062921+0000 2024-03-25T18:09:36.063 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:09:36.062921+0000 2024-03-25T18:09:36.063 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:36.063 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:36.063 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:36.063 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:09:36.063 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:09:36.063 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:36.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.062+0000 7f118f772640 1 Processor -- start 2024-03-25T18:09:36.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.062+0000 7f118f772640 1 -- start start 2024-03-25T18:09:36.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.062+0000 7f118f772640 10 monclient: init 2024-03-25T18:09:36.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.063+0000 7f118f772640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:36.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.063+0000 7f118f772640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:09:36.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.063+0000 7f118f772640 10 monclient(hunting): picked mon.noname-a con 0x7f11880b4700 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:36.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.063+0000 7f118f772640 10 monclient(hunting): picked mon.noname-c con 0x7f1188142a20 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:36.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.063+0000 7f118f772640 10 monclient(hunting): picked mon.noname-b con 0x7f11880a96b0 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:36.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.063+0000 7f118f772640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11880a9d60 con 0x7f11880b4700 2024-03-25T18:09:36.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.063+0000 7f118f772640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118813ff10 con 0x7f1188142a20 2024-03-25T18:09:36.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.063+0000 7f118f772640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11880aa1f0 con 0x7f11880a96b0 2024-03-25T18:09:36.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.063+0000 7f118f772640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:36.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.063+0000 7f118f772640 10 monclient(hunting): authenticate will time out at 6957.300218s 2024-03-25T18:09:36.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.063+0000 7f118cd0d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f1188142a20 0x7f118813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46368/0 (socket says 172.21.15.86:46368) 2024-03-25T18:09:36.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.063+0000 7f118cd0d640 1 -- 172.21.15.86:0/2959627308 learned_addr learned my addr 172.21.15.86:0/2959627308 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-25T18:09:36.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.065+0000 7f117ffff640 1 -- 172.21.15.86:0/2959627308 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176675639 0 0) 0x7f11880a9d60 con 0x7f11880b4700 2024-03-25T18:09:36.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.065+0000 7f117ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:36.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.065+0000 7f117ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:36.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.065+0000 7f117ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:36.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.065+0000 7f117ffff640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:09:36.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.065+0000 7f117ffff640 1 -- 172.21.15.86:0/2959627308 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1174003520 con 0x7f11880b4700 2024-03-25T18:09:36.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.066+0000 7f117ffff640 1 -- 172.21.15.86:0/2959627308 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141047138 0 0) 0x7f118813ff10 con 0x7f1188142a20 2024-03-25T18:09:36.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.066+0000 7f117ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:36.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.066+0000 7f117ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:36.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.066+0000 7f117ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:36.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.066+0000 7f117ffff640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:09:36.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.066+0000 7f117ffff640 1 -- 172.21.15.86:0/2959627308 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11880a9d60 con 0x7f1188142a20 2024-03-25T18:09:36.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.067+0000 7f117ffff640 1 -- 172.21.15.86:0/2959627308 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901465785 0 0) 0x7f11880aa1f0 con 0x7f11880a96b0 2024-03-25T18:09:36.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.067+0000 7f117ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:36.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.067+0000 7f117ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:36.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.067+0000 7f117ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:36.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.067+0000 7f117ffff640 10 monclient(hunting): my global_id is 4115 2024-03-25T18:09:36.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.067+0000 7f117ffff640 1 -- 172.21.15.86:0/2959627308 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f118813ff10 con 0x7f11880a96b0 2024-03-25T18:09:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.069+0000 7f117ffff640 1 -- 172.21.15.86:0/2959627308 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 979472548 0 0) 0x7f1174003520 con 0x7f11880b4700 2024-03-25T18:09:36.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.069+0000 7f117ffff640 1 -- 172.21.15.86:0/2959627308 >> v1:172.21.15.64:6790/0 conn(0x7f11880a96b0 legacy=0x7f118813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:36.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.069+0000 7f117ffff640 1 -- 172.21.15.86:0/2959627308 >> v1:172.21.15.86:6789/0 conn(0x7f1188142a20 legacy=0x7f118813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:36.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.069+0000 7f117ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:09:36.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.069+0000 7f117ffff640 1 monclient: found mon.noname-a 2024-03-25T18:09:36.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.069+0000 7f117ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:36.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.069+0000 7f117ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:09:36.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.069+0000 7f117ffff640 1 -- 172.21.15.86:0/2959627308 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11880b4ae0 con 0x7f11880b4700 2024-03-25T18:09:36.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.069+0000 7f117ffff640 10 monclient: _finish_auth 0 2024-03-25T18:09:36.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.069+0000 7f117ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:09:36.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.069+0000 7f117ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:36.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.069+0000 7f117ffff640 1 -- 172.21.15.86:0/2959627308 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1178001690 con 0x7f11880b4700 2024-03-25T18:09:36.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.069+0000 7f117ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:36.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.069+0000 7f117ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:36.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.069+0000 7f117ffff640 10 monclient: dump: 2024-03-25T18:09:36.071 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:36.072 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.069+0000 7f118f772640 5 monclient: authenticate success, global_id 4116 2024-03-25T18:09:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.069+0000 7f118f772640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:36.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.098+0000 7f117ffff640 1 -- 172.21.15.86:0/2959627308 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11780017f0 con 0x7f11880b4700 2024-03-25T18:09:36.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.098+0000 7f117ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:36.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.098+0000 7f117ffff640 1 -- 172.21.15.86:0/2959627308 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1178001c00 con 0x7f11880b4700 2024-03-25T18:09:36.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.098+0000 7f117ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:36.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.098+0000 7f117ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:36.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.098+0000 7f117ffff640 10 monclient: dump: 2024-03-25T18:09:36.100 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:36.100 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:36.100 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.100 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.100 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:36.100 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:36.100 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:36.100 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:36.100 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:36.100 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:36.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.098+0000 7f118f772640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:36.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.098+0000 7f118f772640 4 set_mon_vals no callback set 2024-03-25T18:09:36.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.099+0000 7f118f772640 10 monclient: shutdown 2024-03-25T18:09:36.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.099+0000 7f118f772640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:36.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.099+0000 7f118f772640 1 -- 172.21.15.86:0/2959627308 >> v1:172.21.15.64:6789/0 conn(0x7f11880b4700 legacy=0x7f1188142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:36.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.099+0000 7f118f772640 1 -- 172.21.15.86:0/2959627308 shutdown_connections 2024-03-25T18:09:36.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.099+0000 7f118f772640 1 -- 172.21.15.86:0/2959627308 >> 172.21.15.86:0/2959627308 conn(0x7f11880668e0 msgr2=0x7f1188066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:36.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.099+0000 7f118f772640 1 -- 172.21.15.86:0/2959627308 shutdown_connections 2024-03-25T18:09:36.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.099+0000 7f118f772640 1 -- 172.21.15.86:0/2959627308 wait complete. 2024-03-25T18:09:36.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.100+0000 7f118f772640 10 monclient: build_initial_monmap 2024-03-25T18:09:36.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.100+0000 7f118f772640 10 monclient: monmap: 2024-03-25T18:09:36.103 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-25T18:09:36.103 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:36.104 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-03-25T18:09:36.104 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-03-25T18:09:36.104 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:36.104 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-25T18:09:36.104 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:36.104 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:09:36.104 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:09:36.104 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:36.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.100+0000 7f118f772640 1 librados: starting msgr at 2024-03-25T18:09:36.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.100+0000 7f118f772640 1 librados: starting objecter 2024-03-25T18:09:36.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.100+0000 7f118f772640 1 Processor -- start 2024-03-25T18:09:36.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.100+0000 7f118f772640 1 -- start start 2024-03-25T18:09:36.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.100+0000 7f118f772640 1 librados: setting wanted keys 2024-03-25T18:09:36.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.100+0000 7f118f772640 1 librados: calling monclient init 2024-03-25T18:09:36.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.100+0000 7f118f772640 10 monclient: init 2024-03-25T18:09:36.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.101+0000 7f118f772640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:36.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.101+0000 7f118f772640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:09:36.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.101+0000 7f118f772640 10 monclient(hunting): picked mon.noname-b con 0x7f11880a96b0 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:36.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.101+0000 7f118f772640 10 monclient(hunting): picked mon.noname-a con 0x7f11880b4700 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:36.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.101+0000 7f118f772640 10 monclient(hunting): picked mon.noname-c con 0x7f1188142a20 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:36.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.101+0000 7f118f772640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11880aba20 con 0x7f11880b4700 2024-03-25T18:09:36.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.101+0000 7f118f772640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11880ad830 con 0x7f11880a96b0 2024-03-25T18:09:36.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.101+0000 7f118f772640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11881db6a0 con 0x7f1188142a20 2024-03-25T18:09:36.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.101+0000 7f118f772640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:36.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.101+0000 7f118f772640 10 monclient(hunting): authenticate will time out at 6957.338229s 2024-03-25T18:09:36.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.101+0000 7f118d50e640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f11880a96b0 0x7f11881d4780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46374/0 (socket says 172.21.15.86:46374) 2024-03-25T18:09:36.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.101+0000 7f118d50e640 1 -- 172.21.15.86:0/177179470 learned_addr learned my addr 172.21.15.86:0/177179470 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-25T18:09:36.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.102+0000 7f117dffb640 10 client.?.objecter ms_handle_connect 0x7f11880a96b0 2024-03-25T18:09:36.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.102+0000 7f117dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:36.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.102+0000 7f117dffb640 10 client.?.objecter ms_handle_connect 0x7f1188142a20 2024-03-25T18:09:36.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.102+0000 7f117dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:36.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.103+0000 7f117dffb640 10 client.?.objecter ms_handle_connect 0x7f11880b4700 2024-03-25T18:09:36.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.103+0000 7f117dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:36.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.104+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611559128 0 0) 0x7f11881db6a0 con 0x7f1188142a20 2024-03-25T18:09:36.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.104+0000 7f117dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:36.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.104+0000 7f117dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:36.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.104+0000 7f117dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:36.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.104+0000 7f117dffb640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:09:36.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.104+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f116c004eb0 con 0x7f1188142a20 2024-03-25T18:09:36.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.105+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311223924 0 0) 0x7f11880aba20 con 0x7f11880b4700 2024-03-25T18:09:36.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.105+0000 7f117dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:36.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.105+0000 7f117dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:36.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.105+0000 7f117dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:36.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.105+0000 7f117dffb640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:09:36.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.105+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11881db6a0 con 0x7f11880b4700 2024-03-25T18:09:36.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.105+0000 7f118cd0d640 1 -- 172.21.15.86:0/177179470 >> v1:172.21.15.64:6789/0 conn(0x7f11880b4700 legacy=0x7f11881da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:09:36.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.105+0000 7f118cd0d640 1 -- 172.21.15.86:0/177179470 >> v1:172.21.15.64:6789/0 conn(0x7f11880b4700 legacy=0x7f11881da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:09:36.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.105+0000 7f118cd0d640 1 --1- 172.21.15.86:0/177179470 >> v1:172.21.15.64:6789/0 conn(0x7f11880b4700 0x7f11881da880 :-1 s=OPENED pgs=23 cs=1 l=1).handle_message read tag failed 2024-03-25T18:09:36.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.105+0000 7f118cd0d640 1 --1- 172.21.15.86:0/177179470 >> v1:172.21.15.64:6789/0 conn(0x7f11880b4700 0x7f11881da880 :-1 s=OPENED pgs=23 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:09:36.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.105+0000 7f117dffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.64:6789/0 2024-03-25T18:09:36.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.106+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001831627 0 0) 0x7f11880ad830 con 0x7f11880a96b0 2024-03-25T18:09:36.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.106+0000 7f117dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:36.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.106+0000 7f117dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:36.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.106+0000 7f117dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:36.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.106+0000 7f117dffb640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:09:36.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.106+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11880aba20 con 0x7f11880a96b0 2024-03-25T18:09:36.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.109+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 553318253 0 0) 0x7f116c004eb0 con 0x7f1188142a20 2024-03-25T18:09:36.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.109+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11880ad830 con 0x7f1188142a20 2024-03-25T18:09:36.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.109+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1180001bd0 con 0x7f1188142a20 2024-03-25T18:09:36.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.109+0000 7f117dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:36.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.109+0000 7f117dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:36.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.109+0000 7f117dffb640 10 monclient(hunting): dump: 2024-03-25T18:09:36.111 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:36.111 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:36.111 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.111 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.111 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:36.111 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:36.111 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:36.111 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:36.111 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:36.112 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:36.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.109+0000 7f117dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:36.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.111+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4034155310 0 0) 0x7f11880aba20 con 0x7f11880a96b0 2024-03-25T18:09:36.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.111+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f116c004eb0 con 0x7f11880a96b0 2024-03-25T18:09:36.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.111+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1178001b00 con 0x7f11880a96b0 2024-03-25T18:09:36.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.111+0000 7f117dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.111+0000 7f117dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.111+0000 7f117dffb640 10 monclient(hunting): dump: 2024-03-25T18:09:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:36.113 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.111+0000 7f118d50e640 1 -- 172.21.15.86:0/177179470 >> v1:172.21.15.86:6789/0 conn(0x7f11880a96b0 legacy=0x7f11881d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T18:09:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.111+0000 7f118d50e640 1 -- 172.21.15.86:0/177179470 >> v1:172.21.15.86:6789/0 conn(0x7f11880a96b0 legacy=0x7f11881d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:09:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.111+0000 7f118d50e640 1 --1- 172.21.15.86:0/177179470 >> v1:172.21.15.86:6789/0 conn(0x7f11880a96b0 0x7f11881d4780 :-1 s=OPENED pgs=24 cs=1 l=1).handle_message read tag failed 2024-03-25T18:09:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.111+0000 7f118d50e640 1 --1- 172.21.15.86:0/177179470 >> v1:172.21.15.86:6789/0 conn(0x7f11880a96b0 0x7f11881d4780 :-1 s=OPENED pgs=24 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:09:36.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.111+0000 7f117dffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.86:6789/0 2024-03-25T18:09:36.114 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:36.112+0000 7fb779810040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:09:36.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.112+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 936866000 0 0) 0x7f11880ad830 con 0x7f1188142a20 2024-03-25T18:09:36.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.112+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 >> v1:172.21.15.86:6789/0 conn(0x7f11880a96b0 legacy=0x7f11881d4780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:09:36.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.112+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 >> v1:172.21.15.64:6789/0 conn(0x7f11880b4700 legacy=0x7f11881da880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:09:36.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.112+0000 7f117dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:09:36.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.112+0000 7f117dffb640 1 monclient: found mon.c 2024-03-25T18:09:36.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.112+0000 7f117dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:36.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.112+0000 7f117dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:36.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.112+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11881d4e90 con 0x7f1188142a20 2024-03-25T18:09:36.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.112+0000 7f117dffb640 10 monclient: _finish_auth 0 2024-03-25T18:09:36.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.112+0000 7f117dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:09:36.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.112+0000 7f117dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:36.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.112+0000 7f118f772640 5 monclient: authenticate success, global_id 4118 2024-03-25T18:09:36.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.112+0000 7f118f772640 10 monclient: _renew_subs 2024-03-25T18:09:36.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.112+0000 7f118f772640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:36.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.112+0000 7f118f772640 1 -- 172.21.15.86:0/177179470 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f118813ba80 con 0x7f1188142a20 2024-03-25T18:09:36.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.113+0000 7f118f772640 10 client.4118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:36.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.113+0000 7f118f772640 10 monclient: _renew_subs 2024-03-25T18:09:36.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.113+0000 7f118f772640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:36.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.113+0000 7f118f772640 1 -- 172.21.15.86:0/177179470 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11880ac790 con 0x7f1188142a20 2024-03-25T18:09:36.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.113+0000 7f118f772640 1 librados: init done 2024-03-25T18:09:36.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.113+0000 7f118f772640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:36.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.113+0000 7f118f772640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:36.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.113+0000 7f118f772640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:36.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.113+0000 7f118f772640 1 -- 172.21.15.86:0/177179470 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11880ab880 con 0x7f1188142a20 2024-03-25T18:09:36.170 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:36.169+0000 7fb779810040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:09:36.213 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:36.213+0000 7f7080038040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:09:36.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.216+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1180002060 con 0x7f1188142a20 2024-03-25T18:09:36.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.216+0000 7f117dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:36.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.216+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1180003990 con 0x7f1188142a20 2024-03-25T18:09:36.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.216+0000 7f117dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:36.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.216+0000 7f117dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:36.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.216+0000 7f117dffb640 10 monclient: dump: 2024-03-25T18:09:36.217 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:36.217 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:36.217 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.217 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.217 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:36.217 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:36.218 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:36.218 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:36.218 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:36.218 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:36.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.216+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1180003f40 con 0x7f1188142a20 2024-03-25T18:09:36.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.216+0000 7f117effd640 4 set_mon_vals no callback set 2024-03-25T18:09:36.218 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:36.216+0000 7fb779810040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:09:36.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.217+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 905920974 0 0) 0x7f1180005410 con 0x7f1188142a20 2024-03-25T18:09:36.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.217+0000 7f117dffb640 10 client.4118.objecter ms_dispatch 0x7f1188000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:09:36.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.217+0000 7f117dffb640 3 client.4118.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:09:36.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.217+0000 7f117dffb640 3 client.4118.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:09:36.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.218+0000 7f117dffb640 20 client.4118.objecter dump_active .. 0 homeless 2024-03-25T18:09:36.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.218+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f11800049c0 con 0x7f1188142a20 2024-03-25T18:09:36.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.218+0000 7f117dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:36.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.218+0000 7f117dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:36.269 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:36.268+0000 7f7080038040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:09:36.275 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:36.274+0000 7fb779810040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:09:36.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.291+0000 7f118f772640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b7d57e83-d493-43bf-8e30-09bcea0e9df8", "id": 0}] 2024-03-25T18:09:36.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.291+0000 7f118f772640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b7d57e83-d493-43bf-8e30-09bcea0e9df8", "id": 0}] 2024-03-25T18:09:36.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.291+0000 7f118f772640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:36.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.291+0000 7f118f772640 1 -- 172.21.15.86:0/177179470 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd new", "uuid": "b7d57e83-d493-43bf-8e30-09bcea0e9df8", "id": 0} v 0) v1 -- 0x7f118804f730 con 0x7f1188142a20 2024-03-25T18:09:36.315 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:36.314+0000 7f7080038040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:09:36.319 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:36.319+0000 7fb779810040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:09:36.368 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:36.367+0000 7f7080038040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:09:36.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.381+0000 7f117dffb640 1 -- 172.21.15.86:0/177179470 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b7d57e83-d493-43bf-8e30-09bcea0e9df8", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1097378102 0 732707515) 0x7f1180004380 con 0x7f1188142a20 2024-03-25T18:09:36.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.382+0000 7f117dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b7d57e83-d493-43bf-8e30-09bcea0e9df8", "id": 0}] 2024-03-25T18:09:36.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.382+0000 7f117dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:36.383 INFO:teuthology.orchestra.run.smithi086.stdout:0 2024-03-25T18:09:36.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.382+0000 7f118f772640 10 librados: watch_flush enter 2024-03-25T18:09:36.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.382+0000 7f118f772640 10 librados: watch_flush exit 2024-03-25T18:09:36.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.382+0000 7f118f772640 20 client.4118.objecter shutdown clearing up homeless session... 2024-03-25T18:09:36.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.382+0000 7f118f772640 10 client.4118.objecter successfully canceled tick 2024-03-25T18:09:36.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.383+0000 7f118f772640 10 monclient: shutdown 2024-03-25T18:09:36.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.383+0000 7f118f772640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:36.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.383+0000 7f118f772640 1 -- 172.21.15.86:0/177179470 >> v1:172.21.15.64:6790/0 conn(0x7f1188142a20 legacy=0x7f11881daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:36.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.383+0000 7f118f772640 1 -- 172.21.15.86:0/177179470 shutdown_connections 2024-03-25T18:09:36.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.383+0000 7f118f772640 1 -- 172.21.15.86:0/177179470 >> 172.21.15.86:0/177179470 conn(0x7f11880668e0 msgr2=0x7f11880b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:36.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.383+0000 7f118f772640 1 -- 172.21.15.86:0/177179470 shutdown_connections 2024-03-25T18:09:36.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.383+0000 7f118f772640 1 -- 172.21.15.86:0/177179470 wait complete. 2024-03-25T18:09:36.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.383+0000 7f118f772640 1 librados: shutdown 2024-03-25T18:09:36.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 739d7438-1775-493c-9156-4f5411c63dd5 1 2024-03-25T18:09:36.414 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:36.413+0000 7f7080038040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:09:36.443 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:36.443+0000 7fb779810040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:09:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.493+0000 7fde85aa8640 10 monclient: get_monmap_and_config 2024-03-25T18:09:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.493+0000 7fde85aa8640 10 monclient: build_initial_monmap 2024-03-25T18:09:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.493+0000 7fde85aa8640 10 monclient: monmap: 2024-03-25T18:09:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-25T18:09:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:09:36.494322+0000 2024-03-25T18:09:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:09:36.494322+0000 2024-03-25T18:09:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:36.495 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:09:36.495 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:09:36.495 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:36.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.493+0000 7fde85aa8640 1 Processor -- start 2024-03-25T18:09:36.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.494+0000 7fde85aa8640 1 -- start start 2024-03-25T18:09:36.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.494+0000 7fde85aa8640 10 monclient: init 2024-03-25T18:09:36.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.494+0000 7fde85aa8640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.494+0000 7fde85aa8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:09:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.494+0000 7fde85aa8640 10 monclient(hunting): picked mon.noname-c con 0x7fde800b4700 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.494+0000 7fde85aa8640 10 monclient(hunting): picked mon.noname-a con 0x7fde80142a20 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.494+0000 7fde85aa8640 10 monclient(hunting): picked mon.noname-b con 0x7fde800a96b0 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.494+0000 7fde85aa8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde800a9d60 con 0x7fde80142a20 2024-03-25T18:09:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.494+0000 7fde85aa8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8013ff10 con 0x7fde800b4700 2024-03-25T18:09:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.494+0000 7fde85aa8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde800aa1f0 con 0x7fde800a96b0 2024-03-25T18:09:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.494+0000 7fde85aa8640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.494+0000 7fde85aa8640 10 monclient(hunting): authenticate will time out at 6957.731507s 2024-03-25T18:09:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.495+0000 7fde7f7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fde800b4700 0x7fde80142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46380/0 (socket says 172.21.15.86:46380) 2024-03-25T18:09:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.495+0000 7fde7f7fe640 1 -- 172.21.15.86:0/1182497606 learned_addr learned my addr 172.21.15.86:0/1182497606 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-25T18:09:36.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.497+0000 7fde7e7fc640 1 -- 172.21.15.86:0/1182497606 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3509343962 0 0) 0x7fde800a9d60 con 0x7fde80142a20 2024-03-25T18:09:36.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.497+0000 7fde7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:36.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.497+0000 7fde7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.497+0000 7fde7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.497+0000 7fde7e7fc640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:09:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.497+0000 7fde7e7fc640 1 -- 172.21.15.86:0/1182497606 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde68003520 con 0x7fde80142a20 2024-03-25T18:09:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.497+0000 7fde7e7fc640 1 -- 172.21.15.86:0/1182497606 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550532535 0 0) 0x7fde8013ff10 con 0x7fde800b4700 2024-03-25T18:09:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.497+0000 7fde7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.497+0000 7fde7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.497+0000 7fde7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.497+0000 7fde7e7fc640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:09:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.497+0000 7fde7e7fc640 1 -- 172.21.15.86:0/1182497606 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde800a9d60 con 0x7fde800b4700 2024-03-25T18:09:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.497+0000 7fde7e7fc640 1 -- 172.21.15.86:0/1182497606 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399717293 0 0) 0x7fde800aa1f0 con 0x7fde800a96b0 2024-03-25T18:09:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.497+0000 7fde7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.497+0000 7fde7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.497+0000 7fde7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.497+0000 7fde7e7fc640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:09:36.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.497+0000 7fde7e7fc640 1 -- 172.21.15.86:0/1182497606 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde8013ff10 con 0x7fde800a96b0 2024-03-25T18:09:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.500+0000 7fde7e7fc640 1 -- 172.21.15.86:0/1182497606 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2413224424 0 0) 0x7fde68003520 con 0x7fde80142a20 2024-03-25T18:09:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.500+0000 7fde7e7fc640 1 -- 172.21.15.86:0/1182497606 >> v1:172.21.15.64:6790/0 conn(0x7fde800a96b0 legacy=0x7fde8013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.500+0000 7fde7e7fc640 1 -- 172.21.15.86:0/1182497606 >> v1:172.21.15.86:6789/0 conn(0x7fde800b4700 legacy=0x7fde80142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.500+0000 7fde7e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:09:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.500+0000 7fde7e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:09:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.500+0000 7fde7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.500+0000 7fde7e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:09:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.500+0000 7fde7e7fc640 1 -- 172.21.15.86:0/1182497606 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde800b4ae0 con 0x7fde80142a20 2024-03-25T18:09:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.500+0000 7fde7e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:09:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.500+0000 7fde7e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:09:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.500+0000 7fde7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.500+0000 7fde85aa8640 5 monclient: authenticate success, global_id 4122 2024-03-25T18:09:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.500+0000 7fde85aa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.501+0000 7fde7e7fc640 1 -- 172.21.15.86:0/1182497606 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fde74001650 con 0x7fde80142a20 2024-03-25T18:09:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.501+0000 7fde7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.501+0000 7fde7e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.501+0000 7fde7e7fc640 10 monclient: dump: 2024-03-25T18:09:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:36.504 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:36.504 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:36.504 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:36.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.501+0000 7fde85aa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:36.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.503+0000 7fde7e7fc640 1 -- 172.21.15.86:0/1182497606 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde740018a0 con 0x7fde80142a20 2024-03-25T18:09:36.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.503+0000 7fde7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.503+0000 7fde7e7fc640 1 -- 172.21.15.86:0/1182497606 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fde74001c30 con 0x7fde80142a20 2024-03-25T18:09:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.503+0000 7fde7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.503+0000 7fde7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.503+0000 7fde7e7fc640 10 monclient: dump: 2024-03-25T18:09:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:36.507 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.503+0000 7fde85aa8640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.503+0000 7fde85aa8640 4 set_mon_vals no callback set 2024-03-25T18:09:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.504+0000 7fde85aa8640 10 monclient: shutdown 2024-03-25T18:09:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.504+0000 7fde85aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.504+0000 7fde85aa8640 1 -- 172.21.15.86:0/1182497606 >> v1:172.21.15.64:6789/0 conn(0x7fde80142a20 legacy=0x7fde8013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.504+0000 7fde85aa8640 1 -- 172.21.15.86:0/1182497606 shutdown_connections 2024-03-25T18:09:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.504+0000 7fde85aa8640 1 -- 172.21.15.86:0/1182497606 >> 172.21.15.86:0/1182497606 conn(0x7fde800668e0 msgr2=0x7fde80066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.504+0000 7fde85aa8640 1 -- 172.21.15.86:0/1182497606 shutdown_connections 2024-03-25T18:09:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.504+0000 7fde85aa8640 1 -- 172.21.15.86:0/1182497606 wait complete. 2024-03-25T18:09:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.504+0000 7fde85aa8640 10 monclient: build_initial_monmap 2024-03-25T18:09:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.504+0000 7fde85aa8640 10 monclient: monmap: 2024-03-25T18:09:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-25T18:09:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-03-25T18:09:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-03-25T18:09:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-25T18:09:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:09:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:09:36.508 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.505+0000 7fde85aa8640 1 librados: starting msgr at 2024-03-25T18:09:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.505+0000 7fde85aa8640 1 librados: starting objecter 2024-03-25T18:09:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.505+0000 7fde85aa8640 1 Processor -- start 2024-03-25T18:09:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.505+0000 7fde85aa8640 1 -- start start 2024-03-25T18:09:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.505+0000 7fde85aa8640 1 librados: setting wanted keys 2024-03-25T18:09:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.505+0000 7fde85aa8640 1 librados: calling monclient init 2024-03-25T18:09:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.505+0000 7fde85aa8640 10 monclient: init 2024-03-25T18:09:36.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.505+0000 7fde85aa8640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:36.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.505+0000 7fde85aa8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:09:36.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.505+0000 7fde85aa8640 10 monclient(hunting): picked mon.noname-a con 0x7fde800a96b0 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:36.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.505+0000 7fde85aa8640 10 monclient(hunting): picked mon.noname-b con 0x7fde800b4700 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:36.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.505+0000 7fde85aa8640 10 monclient(hunting): picked mon.noname-c con 0x7fde80142a20 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:36.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.505+0000 7fde85aa8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde800ae280 con 0x7fde800a96b0 2024-03-25T18:09:36.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.505+0000 7fde85aa8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde800ae560 con 0x7fde800b4700 2024-03-25T18:09:36.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.505+0000 7fde85aa8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde800acb00 con 0x7fde80142a20 2024-03-25T18:09:36.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.505+0000 7fde85aa8640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:36.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.505+0000 7fde85aa8640 10 monclient(hunting): authenticate will time out at 6957.742857s 2024-03-25T18:09:36.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.506+0000 7fde7f7fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fde800a96b0 0x7fde800ac170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.86:42656/0 (socket says 172.21.15.86:42656) 2024-03-25T18:09:36.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.506+0000 7fde7effd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fde800b4700 0x7fde800ae980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46384/0 (socket says 172.21.15.86:46384) 2024-03-25T18:09:36.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.506+0000 7fde7f7fe640 1 -- 172.21.15.86:0/2812493827 learned_addr learned my addr 172.21.15.86:0/2812493827 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-25T18:09:36.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.506+0000 7fde67fff640 10 client.?.objecter ms_handle_connect 0x7fde800b4700 2024-03-25T18:09:36.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.506+0000 7fde67fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:36.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.506+0000 7fde67fff640 10 client.?.objecter ms_handle_connect 0x7fde800a96b0 2024-03-25T18:09:36.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.506+0000 7fde67fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.508+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027360813 0 0) 0x7fde800ae560 con 0x7fde800b4700 2024-03-25T18:09:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.508+0000 7fde67fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.508+0000 7fde67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.508+0000 7fde67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.508+0000 7fde67fff640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:09:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.508+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde6c004eb0 con 0x7fde800b4700 2024-03-25T18:09:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.509+0000 7fde67fff640 10 client.?.objecter ms_handle_connect 0x7fde80142a20 2024-03-25T18:09:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.509+0000 7fde67fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.509+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899307054 0 0) 0x7fde800ae280 con 0x7fde800a96b0 2024-03-25T18:09:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.509+0000 7fde67fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.509+0000 7fde67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.509+0000 7fde67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.509+0000 7fde67fff640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:09:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.509+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde800ae560 con 0x7fde800a96b0 2024-03-25T18:09:36.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.511+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290428548 0 0) 0x7fde800acb00 con 0x7fde80142a20 2024-03-25T18:09:36.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.511+0000 7fde67fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:36.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.511+0000 7fde67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:36.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.511+0000 7fde67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:36.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.511+0000 7fde67fff640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:09:36.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.511+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde800ae280 con 0x7fde80142a20 2024-03-25T18:09:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.514+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2005529477 0 0) 0x7fde800ae560 con 0x7fde800a96b0 2024-03-25T18:09:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.514+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde800acb00 con 0x7fde800a96b0 2024-03-25T18:09:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.514+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fde70001930 con 0x7fde800a96b0 2024-03-25T18:09:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.514+0000 7fde67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.514+0000 7fde67fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.514+0000 7fde67fff640 10 monclient(hunting): dump: 2024-03-25T18:09:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:36.516 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.514+0000 7fde67fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.514+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3746094322 0 0) 0x7fde6c004eb0 con 0x7fde800b4700 2024-03-25T18:09:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.514+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde800ae560 con 0x7fde800b4700 2024-03-25T18:09:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.514+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fde74001bb0 con 0x7fde800b4700 2024-03-25T18:09:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.514+0000 7fde67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.514+0000 7fde67fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.514+0000 7fde67fff640 10 monclient(hunting): dump: 2024-03-25T18:09:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:36.517 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.517 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.517 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:36.517 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:36.517 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:36.517 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:36.517 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:36.517 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.516+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 719427917 0 0) 0x7fde800ae280 con 0x7fde80142a20 2024-03-25T18:09:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.516+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde6c004eb0 con 0x7fde80142a20 2024-03-25T18:09:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.516+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fde78001bd0 con 0x7fde80142a20 2024-03-25T18:09:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.516+0000 7fde67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.516+0000 7fde67fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.516+0000 7fde67fff640 10 monclient(hunting): dump: 2024-03-25T18:09:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:36.519 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.520 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.520 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:36.520 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:36.520 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:36.520 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:36.520 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:36.520 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:36.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1252887415 0 0) 0x7fde800ae560 con 0x7fde800b4700 2024-03-25T18:09:36.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 >> v1:172.21.15.64:6790/0 conn(0x7fde80142a20 legacy=0x7fde801e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:36.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 >> v1:172.21.15.64:6789/0 conn(0x7fde800a96b0 legacy=0x7fde800ac170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:36.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde67fff640 10 monclient: _finish_hunting 0 2024-03-25T18:09:36.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde67fff640 1 monclient: found mon.b 2024-03-25T18:09:36.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:36.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde67fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:36.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde800af090 con 0x7fde800b4700 2024-03-25T18:09:36.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde67fff640 10 monclient: _finish_auth 0 2024-03-25T18:09:36.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde67fff640 10 monclient: _check_auth_tickets 2024-03-25T18:09:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde85aa8640 5 monclient: authenticate success, global_id 4126 2024-03-25T18:09:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde85aa8640 10 monclient: _renew_subs 2024-03-25T18:09:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde85aa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde85aa8640 1 -- 172.21.15.86:0/2812493827 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8013af90 con 0x7fde800b4700 2024-03-25T18:09:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde85aa8640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde85aa8640 10 monclient: _renew_subs 2024-03-25T18:09:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde85aa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde85aa8640 1 -- 172.21.15.86:0/2812493827 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde800ad200 con 0x7fde800b4700 2024-03-25T18:09:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.517+0000 7fde85aa8640 1 librados: init done 2024-03-25T18:09:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.518+0000 7fde85aa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.518+0000 7fde85aa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.518+0000 7fde85aa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:36.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.518+0000 7fde85aa8640 1 -- 172.21.15.86:0/2812493827 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde8013b180 con 0x7fde800b4700 2024-03-25T18:09:36.522 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:36.517+0000 7fb779810040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:09:36.540 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:36.539+0000 7f7080038040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:09:36.566 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:36.565+0000 7fb779810040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:09:36.611 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:36.611+0000 7f7080038040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:09:36.620 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:36.620+0000 7fb779810040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:09:36.659 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:36.658+0000 7f7080038040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:09:36.670 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:36.669+0000 7fb779810040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:09:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.692+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde74003960 con 0x7fde800b4700 2024-03-25T18:09:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.692+0000 7fde67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.692+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fde74003bd0 con 0x7fde800b4700 2024-03-25T18:09:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.692+0000 7fde67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.692+0000 7fde67fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.692+0000 7fde67fff640 10 monclient: dump: 2024-03-25T18:09:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:36.695 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.692+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fde74004180 con 0x7fde800b4700 2024-03-25T18:09:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.692+0000 7fde7dffb640 4 set_mon_vals no callback set 2024-03-25T18:09:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.692+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 660771649 0 0) 0x7fde740065c0 con 0x7fde800b4700 2024-03-25T18:09:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.692+0000 7fde67fff640 10 client.4126.objecter ms_dispatch 0x7fde80000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:09:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.692+0000 7fde67fff640 3 client.4126.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:09:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.692+0000 7fde67fff640 3 client.4126.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:09:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.692+0000 7fde67fff640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-25T18:09:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.693+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fde740069a0 con 0x7fde800b4700 2024-03-25T18:09:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.693+0000 7fde67fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.693+0000 7fde67fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:36.715 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:36.715+0000 7f7080038040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:09:36.766 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:36.765+0000 7f7080038040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:09:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.784+0000 7fde85aa8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "739d7438-1775-493c-9156-4f5411c63dd5", "id": 1}] 2024-03-25T18:09:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.784+0000 7fde85aa8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "739d7438-1775-493c-9156-4f5411c63dd5", "id": 1}] 2024-03-25T18:09:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.784+0000 7fde85aa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.784+0000 7fde85aa8640 1 -- 172.21.15.86:0/2812493827 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd new", "uuid": "739d7438-1775-493c-9156-4f5411c63dd5", "id": 1} v 0) v1 -- 0x7fde800ad500 con 0x7fde800b4700 2024-03-25T18:09:36.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.923+0000 7fde67fff640 1 -- 172.21.15.86:0/2812493827 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "739d7438-1775-493c-9156-4f5411c63dd5", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3602172688 0 940483276) 0x7fde74004430 con 0x7fde800b4700 2024-03-25T18:09:36.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.923+0000 7fde67fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "739d7438-1775-493c-9156-4f5411c63dd5", "id": 1}] 2024-03-25T18:09:36.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.923+0000 7fde67fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:36.924 INFO:teuthology.orchestra.run.smithi086.stdout:1 2024-03-25T18:09:36.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.924+0000 7fde85aa8640 10 librados: watch_flush enter 2024-03-25T18:09:36.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.924+0000 7fde85aa8640 10 librados: watch_flush exit 2024-03-25T18:09:36.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.924+0000 7fde85aa8640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-25T18:09:36.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.924+0000 7fde85aa8640 10 client.4126.objecter successfully canceled tick 2024-03-25T18:09:36.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.924+0000 7fde85aa8640 10 monclient: shutdown 2024-03-25T18:09:36.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.924+0000 7fde85aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:36.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.924+0000 7fde85aa8640 1 -- 172.21.15.86:0/2812493827 >> v1:172.21.15.86:6789/0 conn(0x7fde800b4700 legacy=0x7fde800ae980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:36.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.924+0000 7fde85aa8640 1 -- 172.21.15.86:0/2812493827 shutdown_connections 2024-03-25T18:09:36.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.924+0000 7fde85aa8640 1 -- 172.21.15.86:0/2812493827 >> 172.21.15.86:0/2812493827 conn(0x7fde800668e0 msgr2=0x7fde800b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:36.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.924+0000 7fde85aa8640 1 -- 172.21.15.86:0/2812493827 shutdown_connections 2024-03-25T18:09:36.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.924+0000 7fde85aa8640 1 -- 172.21.15.86:0/2812493827 wait complete. 2024-03-25T18:09:36.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:36.924+0000 7fde85aa8640 1 librados: shutdown 2024-03-25T18:09:36.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new ca83a5b9-4e42-4f79-907f-452b287a3324 2 2024-03-25T18:09:37.076 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:37.075+0000 7fb779810040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:09:37.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.075+0000 7f1c4afe6640 10 monclient: get_monmap_and_config 2024-03-25T18:09:37.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.075+0000 7f1c4afe6640 10 monclient: build_initial_monmap 2024-03-25T18:09:37.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.075+0000 7f1c4afe6640 10 monclient: monmap: 2024-03-25T18:09:37.076 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-25T18:09:37.077 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.077 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:09:37.076592+0000 2024-03-25T18:09:37.077 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:09:37.076592+0000 2024-03-25T18:09:37.077 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:37.077 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.077 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:37.077 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:09:37.077 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:09:37.077 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.076+0000 7f1c4afe6640 1 Processor -- start 2024-03-25T18:09:37.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.076+0000 7f1c4afe6640 1 -- start start 2024-03-25T18:09:37.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.076+0000 7f1c4afe6640 10 monclient: init 2024-03-25T18:09:37.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.076+0000 7f1c4afe6640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:37.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.076+0000 7f1c4afe6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.076+0000 7f1c4afe6640 10 monclient(hunting): picked mon.noname-c con 0x7f1c440b4700 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.076+0000 7f1c4afe6640 10 monclient(hunting): picked mon.noname-b con 0x7f1c44142a20 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.076+0000 7f1c4afe6640 10 monclient(hunting): picked mon.noname-a con 0x7f1c440a96b0 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.076+0000 7f1c4afe6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c440a9d60 con 0x7f1c440a96b0 2024-03-25T18:09:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.076+0000 7f1c4afe6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4413ff10 con 0x7f1c440b4700 2024-03-25T18:09:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.076+0000 7f1c4afe6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c440aa1f0 con 0x7f1c44142a20 2024-03-25T18:09:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.076+0000 7f1c4afe6640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.076+0000 7f1c4afe6640 10 monclient(hunting): authenticate will time out at 6958.313792s 2024-03-25T18:09:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.077+0000 7f1c48d82640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f1c440b4700 0x7f1c44142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46388/0 (socket says 172.21.15.86:46388) 2024-03-25T18:09:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.077+0000 7f1c48d82640 1 -- 172.21.15.86:0/4081618415 learned_addr learned my addr 172.21.15.86:0/4081618415 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-25T18:09:37.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.079+0000 7f1c437fe640 1 -- 172.21.15.86:0/4081618415 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704466346 0 0) 0x7f1c4413ff10 con 0x7f1c440b4700 2024-03-25T18:09:37.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.079+0000 7f1c437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:37.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.079+0000 7f1c437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:37.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.079+0000 7f1c437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:37.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.079+0000 7f1c437fe640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:09:37.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.079+0000 7f1c437fe640 1 -- 172.21.15.86:0/4081618415 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c34003520 con 0x7f1c440b4700 2024-03-25T18:09:37.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.079+0000 7f1c437fe640 1 -- 172.21.15.86:0/4081618415 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156908289 0 0) 0x7f1c440a9d60 con 0x7f1c440a96b0 2024-03-25T18:09:37.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.079+0000 7f1c437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:37.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.079+0000 7f1c437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:37.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.079+0000 7f1c437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:37.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.079+0000 7f1c437fe640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:09:37.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.079+0000 7f1c437fe640 1 -- 172.21.15.86:0/4081618415 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c4413ff10 con 0x7f1c440a96b0 2024-03-25T18:09:37.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.080+0000 7f1c437fe640 1 -- 172.21.15.86:0/4081618415 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483821887 0 0) 0x7f1c440aa1f0 con 0x7f1c44142a20 2024-03-25T18:09:37.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.080+0000 7f1c437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:37.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.080+0000 7f1c437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:37.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.080+0000 7f1c437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:37.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.080+0000 7f1c437fe640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:09:37.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.080+0000 7f1c437fe640 1 -- 172.21.15.86:0/4081618415 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c440a9d60 con 0x7f1c44142a20 2024-03-25T18:09:37.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.082+0000 7f1c437fe640 1 -- 172.21.15.86:0/4081618415 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 881834236 0 0) 0x7f1c34003520 con 0x7f1c440b4700 2024-03-25T18:09:37.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.082+0000 7f1c437fe640 1 -- 172.21.15.86:0/4081618415 >> v1:172.21.15.64:6790/0 conn(0x7f1c44142a20 legacy=0x7f1c4413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.082+0000 7f1c437fe640 1 -- 172.21.15.86:0/4081618415 >> v1:172.21.15.64:6789/0 conn(0x7f1c440a96b0 legacy=0x7f1c4413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.082+0000 7f1c437fe640 10 monclient: _finish_hunting 0 2024-03-25T18:09:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.082+0000 7f1c437fe640 1 monclient: found mon.noname-c 2024-03-25T18:09:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.082+0000 7f1c437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.082+0000 7f1c437fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:09:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.082+0000 7f1c437fe640 1 -- 172.21.15.86:0/4081618415 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c440b4ae0 con 0x7f1c440b4700 2024-03-25T18:09:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.082+0000 7f1c437fe640 10 monclient: _finish_auth 0 2024-03-25T18:09:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.082+0000 7f1c437fe640 10 monclient: _check_auth_tickets 2024-03-25T18:09:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.082+0000 7f1c437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.082+0000 7f1c437fe640 1 -- 172.21.15.86:0/4081618415 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1c38001690 con 0x7f1c440b4700 2024-03-25T18:09:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.082+0000 7f1c437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.082+0000 7f1c437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.082+0000 7f1c437fe640 10 monclient: dump: 2024-03-25T18:09:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:37.085 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.082+0000 7f1c4afe6640 5 monclient: authenticate success, global_id 4129 2024-03-25T18:09:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.082+0000 7f1c4afe6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.084+0000 7f1c437fe640 1 -- 172.21.15.86:0/4081618415 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c380017f0 con 0x7f1c440b4700 2024-03-25T18:09:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.084+0000 7f1c437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.084+0000 7f1c437fe640 1 -- 172.21.15.86:0/4081618415 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1c38001c00 con 0x7f1c440b4700 2024-03-25T18:09:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.084+0000 7f1c4afe6640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.084+0000 7f1c4afe6640 4 set_mon_vals no callback set 2024-03-25T18:09:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.084+0000 7f1c437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.084+0000 7f1c437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.084+0000 7f1c437fe640 10 monclient: dump: 2024-03-25T18:09:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:37.088 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.085+0000 7f1c4afe6640 10 monclient: shutdown 2024-03-25T18:09:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.085+0000 7f1c4afe6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.085+0000 7f1c4afe6640 1 -- 172.21.15.86:0/4081618415 >> v1:172.21.15.86:6789/0 conn(0x7f1c440b4700 legacy=0x7f1c44142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.085+0000 7f1c4afe6640 1 -- 172.21.15.86:0/4081618415 shutdown_connections 2024-03-25T18:09:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.085+0000 7f1c4afe6640 1 -- 172.21.15.86:0/4081618415 >> 172.21.15.86:0/4081618415 conn(0x7f1c440668e0 msgr2=0x7f1c44066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.085+0000 7f1c4afe6640 1 -- 172.21.15.86:0/4081618415 shutdown_connections 2024-03-25T18:09:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.085+0000 7f1c4afe6640 1 -- 172.21.15.86:0/4081618415 wait complete. 2024-03-25T18:09:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.086+0000 7f1c4afe6640 10 monclient: build_initial_monmap 2024-03-25T18:09:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.086+0000 7f1c4afe6640 10 monclient: monmap: 2024-03-25T18:09:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-25T18:09:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-03-25T18:09:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-03-25T18:09:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-25T18:09:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:09:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:09:37.089 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.086+0000 7f1c4afe6640 1 librados: starting msgr at 2024-03-25T18:09:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.086+0000 7f1c4afe6640 1 librados: starting objecter 2024-03-25T18:09:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.086+0000 7f1c4afe6640 1 Processor -- start 2024-03-25T18:09:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.086+0000 7f1c4afe6640 1 -- start start 2024-03-25T18:09:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.086+0000 7f1c4afe6640 1 librados: setting wanted keys 2024-03-25T18:09:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.086+0000 7f1c4afe6640 1 librados: calling monclient init 2024-03-25T18:09:37.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.086+0000 7f1c4afe6640 10 monclient: init 2024-03-25T18:09:37.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.087+0000 7f1c4afe6640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:37.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.087+0000 7f1c4afe6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:09:37.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.087+0000 7f1c4afe6640 10 monclient(hunting): picked mon.noname-c con 0x7f1c440a96b0 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:37.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.087+0000 7f1c4afe6640 10 monclient(hunting): picked mon.noname-a con 0x7f1c440b4700 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:37.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.087+0000 7f1c4afe6640 10 monclient(hunting): picked mon.noname-b con 0x7f1c44142a20 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:37.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.087+0000 7f1c4afe6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c440b1850 con 0x7f1c440b4700 2024-03-25T18:09:37.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.087+0000 7f1c4afe6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c440ad150 con 0x7f1c44142a20 2024-03-25T18:09:37.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.087+0000 7f1c4afe6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c440ad430 con 0x7f1c440a96b0 2024-03-25T18:09:37.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.087+0000 7f1c4afe6640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:37.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.087+0000 7f1c4afe6640 10 monclient(hunting): authenticate will time out at 6958.324104s 2024-03-25T18:09:37.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.087+0000 7f1c49583640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f1c44142a20 0x7f1c441e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46396/0 (socket says 172.21.15.86:46396) 2024-03-25T18:09:37.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.087+0000 7f1c49583640 1 -- 172.21.15.86:0/497351410 learned_addr learned my addr 172.21.15.86:0/497351410 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-25T18:09:37.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.088+0000 7f1c417fa640 10 client.?.objecter ms_handle_connect 0x7f1c440a96b0 2024-03-25T18:09:37.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.088+0000 7f1c417fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:37.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.088+0000 7f1c417fa640 10 client.?.objecter ms_handle_connect 0x7f1c44142a20 2024-03-25T18:09:37.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.088+0000 7f1c417fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:37.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.090+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005721858 0 0) 0x7f1c440ad150 con 0x7f1c44142a20 2024-03-25T18:09:37.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.090+0000 7f1c417fa640 10 monclient(hunting): _negotiate 2024-03-25T18:09:37.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.090+0000 7f1c417fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:37.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.090+0000 7f1c417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:37.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.090+0000 7f1c417fa640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:09:37.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.090+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c28004eb0 con 0x7f1c44142a20 2024-03-25T18:09:37.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.090+0000 7f1c417fa640 10 client.?.objecter ms_handle_connect 0x7f1c440b4700 2024-03-25T18:09:37.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.090+0000 7f1c417fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:37.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.090+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084568977 0 0) 0x7f1c440ad430 con 0x7f1c440a96b0 2024-03-25T18:09:37.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.090+0000 7f1c417fa640 10 monclient(hunting): _negotiate 2024-03-25T18:09:37.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.090+0000 7f1c417fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:37.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.090+0000 7f1c417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:37.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.090+0000 7f1c417fa640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:09:37.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.090+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c440ad150 con 0x7f1c440a96b0 2024-03-25T18:09:37.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.092+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626119267 0 0) 0x7f1c440b1850 con 0x7f1c440b4700 2024-03-25T18:09:37.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.092+0000 7f1c417fa640 10 monclient(hunting): _negotiate 2024-03-25T18:09:37.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.092+0000 7f1c417fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:37.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.092+0000 7f1c417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:37.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.092+0000 7f1c417fa640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:09:37.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.092+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c440ad430 con 0x7f1c440b4700 2024-03-25T18:09:37.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.094+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1985639751 0 0) 0x7f1c28004eb0 con 0x7f1c44142a20 2024-03-25T18:09:37.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.094+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c440b1850 con 0x7f1c44142a20 2024-03-25T18:09:37.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.094+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1c3c001c90 con 0x7f1c44142a20 2024-03-25T18:09:37.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.094+0000 7f1c417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.094+0000 7f1c417fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.094+0000 7f1c417fa640 10 monclient(hunting): dump: 2024-03-25T18:09:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:37.096 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.095+0000 7f1c417fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.096+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 350150978 0 0) 0x7f1c440ad150 con 0x7f1c440a96b0 2024-03-25T18:09:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.096+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c28004eb0 con 0x7f1c440a96b0 2024-03-25T18:09:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.096+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1c38001b00 con 0x7f1c440a96b0 2024-03-25T18:09:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.096+0000 7f1c417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.096+0000 7f1c417fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.096+0000 7f1c417fa640 10 monclient(hunting): dump: 2024-03-25T18:09:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:37.098 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1049760013 0 0) 0x7f1c440ad430 con 0x7f1c440b4700 2024-03-25T18:09:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c440ad150 con 0x7f1c440b4700 2024-03-25T18:09:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1c30001830 con 0x7f1c440b4700 2024-03-25T18:09:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c417fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c417fa640 10 monclient(hunting): dump: 2024-03-25T18:09:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:37.101 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2619318087 0 0) 0x7f1c440b1850 con 0x7f1c44142a20 2024-03-25T18:09:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 >> v1:172.21.15.64:6790/0 conn(0x7f1c440a96b0 legacy=0x7f1c44139db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 >> v1:172.21.15.64:6789/0 conn(0x7f1c440b4700 legacy=0x7f1c440ad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c417fa640 10 monclient: _finish_hunting 0 2024-03-25T18:09:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c417fa640 1 monclient: found mon.b 2024-03-25T18:09:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c417fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c440ac180 con 0x7f1c44142a20 2024-03-25T18:09:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c417fa640 10 monclient: _finish_auth 0 2024-03-25T18:09:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c417fa640 10 monclient: _check_auth_tickets 2024-03-25T18:09:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c4afe6640 5 monclient: authenticate success, global_id 4132 2024-03-25T18:09:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c4afe6640 10 monclient: _renew_subs 2024-03-25T18:09:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c4afe6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c4afe6640 1 -- 172.21.15.86:0/497351410 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c4413b5c0 con 0x7f1c44142a20 2024-03-25T18:09:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c4afe6640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c4afe6640 10 monclient: _renew_subs 2024-03-25T18:09:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c4afe6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c4afe6640 1 -- 172.21.15.86:0/497351410 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c440abaf0 con 0x7f1c44142a20 2024-03-25T18:09:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.097+0000 7f1c4afe6640 1 librados: init done 2024-03-25T18:09:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.098+0000 7f1c4afe6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.098+0000 7f1c4afe6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.098+0000 7f1c4afe6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.098+0000 7f1c4afe6640 1 -- 172.21.15.86:0/497351410 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c4404f730 con 0x7f1c44142a20 2024-03-25T18:09:37.165 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:37.165+0000 7f7080038040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:09:37.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.196+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c3c003970 con 0x7f1c44142a20 2024-03-25T18:09:37.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.196+0000 7f1c417fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:37.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.196+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1c3c003d80 con 0x7f1c44142a20 2024-03-25T18:09:37.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.196+0000 7f1c417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:37.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.196+0000 7f1c417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:37.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.196+0000 7f1c417fa640 10 monclient: dump: 2024-03-25T18:09:37.198 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:37.198 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.198 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.198 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.198 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:37.198 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.198 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:37.198 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:37.199 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.196+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1c3c004330 con 0x7f1c44142a20 2024-03-25T18:09:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.196+0000 7f1c42ffd640 4 set_mon_vals no callback set 2024-03-25T18:09:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.196+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1181362463 0 0) 0x7f1c3c006920 con 0x7f1c44142a20 2024-03-25T18:09:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.196+0000 7f1c417fa640 10 client.4132.objecter ms_dispatch 0x7f1c44000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:09:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.196+0000 7f1c417fa640 3 client.4132.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:09:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.196+0000 7f1c417fa640 3 client.4132.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:09:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.196+0000 7f1c417fa640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-25T18:09:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.196+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1c3c018de0 con 0x7f1c44142a20 2024-03-25T18:09:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.196+0000 7f1c417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.196+0000 7f1c417fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:37.220 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:37.220+0000 7fb779810040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:09:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.269+0000 7f1c4afe6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ca83a5b9-4e42-4f79-907f-452b287a3324", "id": 2}] 2024-03-25T18:09:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.269+0000 7f1c4afe6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ca83a5b9-4e42-4f79-907f-452b287a3324", "id": 2}] 2024-03-25T18:09:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.269+0000 7f1c4afe6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.269+0000 7f1c4afe6640 1 -- 172.21.15.86:0/497351410 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ca83a5b9-4e42-4f79-907f-452b287a3324", "id": 2} v 0) v1 -- 0x7f1c440b7190 con 0x7f1c44142a20 2024-03-25T18:09:37.303 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:37.303+0000 7f7080038040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:09:37.353 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:37.352+0000 7fb779810040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:09:37.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.363+0000 7f1c417fa640 1 -- 172.21.15.86:0/497351410 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ca83a5b9-4e42-4f79-907f-452b287a3324", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3164114611 0 216599125) 0x7f1c3c0045e0 con 0x7f1c44142a20 2024-03-25T18:09:37.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.363+0000 7f1c417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ca83a5b9-4e42-4f79-907f-452b287a3324", "id": 2}] 2024-03-25T18:09:37.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.363+0000 7f1c417fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:37.364 INFO:teuthology.orchestra.run.smithi086.stdout:2 2024-03-25T18:09:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.363+0000 7f1c4afe6640 10 librados: watch_flush enter 2024-03-25T18:09:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.363+0000 7f1c4afe6640 10 librados: watch_flush exit 2024-03-25T18:09:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.363+0000 7f1c4afe6640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-25T18:09:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.363+0000 7f1c4afe6640 10 client.4132.objecter successfully canceled tick 2024-03-25T18:09:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.364+0000 7f1c4afe6640 10 monclient: shutdown 2024-03-25T18:09:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.364+0000 7f1c4afe6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.364+0000 7f1c4afe6640 1 -- 172.21.15.86:0/497351410 >> v1:172.21.15.86:6789/0 conn(0x7f1c44142a20 legacy=0x7f1c441e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.364+0000 7f1c4afe6640 1 -- 172.21.15.86:0/497351410 shutdown_connections 2024-03-25T18:09:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.364+0000 7f1c4afe6640 1 -- 172.21.15.86:0/497351410 >> 172.21.15.86:0/497351410 conn(0x7f1c440668e0 msgr2=0x7f1c441364a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.364+0000 7f1c4afe6640 1 -- 172.21.15.86:0/497351410 shutdown_connections 2024-03-25T18:09:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.364+0000 7f1c4afe6640 1 -- 172.21.15.86:0/497351410 wait complete. 2024-03-25T18:09:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.364+0000 7f1c4afe6640 1 librados: shutdown 2024-03-25T18:09:37.375 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 315a607b-7263-41dc-8587-bd31883b50d6 3 2024-03-25T18:09:37.400 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:37.400+0000 7fb779810040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:09:37.440 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:37.439+0000 7f7080038040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:09:37.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.474+0000 7f0307f54640 10 monclient: get_monmap_and_config 2024-03-25T18:09:37.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.474+0000 7f0307f54640 10 monclient: build_initial_monmap 2024-03-25T18:09:37.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.474+0000 7f0307f54640 10 monclient: monmap: 2024-03-25T18:09:37.475 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-25T18:09:37.475 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.475 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:09:37.475637+0000 2024-03-25T18:09:37.475 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:09:37.475637+0000 2024-03-25T18:09:37.475 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:37.475 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.475 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:37.475 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:09:37.476 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:09:37.476 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.475+0000 7f0307f54640 1 Processor -- start 2024-03-25T18:09:37.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.475+0000 7f0307f54640 1 -- start start 2024-03-25T18:09:37.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.475+0000 7f0307f54640 10 monclient: init 2024-03-25T18:09:37.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.475+0000 7f0307f54640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:37.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.475+0000 7f0307f54640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:37.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.475+0000 7f0307f54640 10 monclient(hunting): picked mon.noname-c con 0x7f03000b4700 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:37.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.475+0000 7f0307f54640 10 monclient(hunting): picked mon.noname-b con 0x7f0300142a20 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:37.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.475+0000 7f0307f54640 10 monclient(hunting): picked mon.noname-a con 0x7f03000a96b0 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:37.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.475+0000 7f0307f54640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03000a9d60 con 0x7f03000a96b0 2024-03-25T18:09:37.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.475+0000 7f0307f54640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030013ff10 con 0x7f03000b4700 2024-03-25T18:09:37.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.475+0000 7f0307f54640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03000aa1f0 con 0x7f0300142a20 2024-03-25T18:09:37.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.475+0000 7f0307f54640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:37.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.475+0000 7f0307f54640 10 monclient(hunting): authenticate will time out at 6958.712844s 2024-03-25T18:09:37.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.476+0000 7f0305cf0640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f03000b4700 0x7f0300142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46408/0 (socket says 172.21.15.86:46408) 2024-03-25T18:09:37.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.476+0000 7f03064f1640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f03000a96b0 0x7f030013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.86:42688/0 (socket says 172.21.15.86:42688) 2024-03-25T18:09:37.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.476+0000 7f0305cf0640 1 -- 172.21.15.86:0/1969903185 learned_addr learned my addr 172.21.15.86:0/1969903185 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-25T18:09:37.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.478+0000 7f0304cee640 1 -- 172.21.15.86:0/1969903185 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254129746 0 0) 0x7f030013ff10 con 0x7f03000b4700 2024-03-25T18:09:37.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.478+0000 7f0304cee640 10 monclient(hunting): _negotiate 2024-03-25T18:09:37.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.478+0000 7f0304cee640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:37.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.478+0000 7f0304cee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:37.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.478+0000 7f0304cee640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:09:37.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.478+0000 7f0304cee640 1 -- 172.21.15.86:0/1969903185 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02ec003520 con 0x7f03000b4700 2024-03-25T18:09:37.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.479+0000 7f0304cee640 1 -- 172.21.15.86:0/1969903185 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21969455 0 0) 0x7f03000a9d60 con 0x7f03000a96b0 2024-03-25T18:09:37.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.479+0000 7f0304cee640 10 monclient(hunting): _negotiate 2024-03-25T18:09:37.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.479+0000 7f0304cee640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:37.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.479+0000 7f0304cee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:37.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.479+0000 7f0304cee640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:09:37.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.479+0000 7f0304cee640 1 -- 172.21.15.86:0/1969903185 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f030013ff10 con 0x7f03000a96b0 2024-03-25T18:09:37.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.479+0000 7f0304cee640 1 -- 172.21.15.86:0/1969903185 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059610341 0 0) 0x7f03000aa1f0 con 0x7f0300142a20 2024-03-25T18:09:37.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.479+0000 7f0304cee640 10 monclient(hunting): _negotiate 2024-03-25T18:09:37.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.479+0000 7f0304cee640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:37.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.479+0000 7f0304cee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:37.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.479+0000 7f0304cee640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:09:37.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.479+0000 7f0304cee640 1 -- 172.21.15.86:0/1969903185 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03000a9d60 con 0x7f0300142a20 2024-03-25T18:09:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.481+0000 7f0304cee640 1 -- 172.21.15.86:0/1969903185 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2934362189 0 0) 0x7f02ec003520 con 0x7f03000b4700 2024-03-25T18:09:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.481+0000 7f0304cee640 1 -- 172.21.15.86:0/1969903185 >> v1:172.21.15.64:6790/0 conn(0x7f0300142a20 legacy=0x7f030013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.481+0000 7f0304cee640 1 -- 172.21.15.86:0/1969903185 >> v1:172.21.15.64:6789/0 conn(0x7f03000a96b0 legacy=0x7f030013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.482+0000 7f0304cee640 10 monclient: _finish_hunting 0 2024-03-25T18:09:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.482+0000 7f0304cee640 1 monclient: found mon.noname-c 2024-03-25T18:09:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.482+0000 7f0304cee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.482+0000 7f0304cee640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:09:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.482+0000 7f0304cee640 1 -- 172.21.15.86:0/1969903185 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03000b4ae0 con 0x7f03000b4700 2024-03-25T18:09:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.482+0000 7f0304cee640 10 monclient: _finish_auth 0 2024-03-25T18:09:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.482+0000 7f0304cee640 10 monclient: _check_auth_tickets 2024-03-25T18:09:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.482+0000 7f0304cee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.482+0000 7f0304cee640 1 -- 172.21.15.86:0/1969903185 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f02fc001690 con 0x7f03000b4700 2024-03-25T18:09:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.482+0000 7f0307f54640 5 monclient: authenticate success, global_id 4135 2024-03-25T18:09:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.482+0000 7f0307f54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:37.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.482+0000 7f0304cee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:37.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.482+0000 7f0304cee640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:37.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.482+0000 7f0304cee640 10 monclient: dump: 2024-03-25T18:09:37.484 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:37.484 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.484 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.484 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.484 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:37.484 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.484 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:37.484 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:37.484 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:37.484 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.482+0000 7f0307f54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:37.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.483+0000 7f0304cee640 1 -- 172.21.15.86:0/1969903185 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02fc0017f0 con 0x7f03000b4700 2024-03-25T18:09:37.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.483+0000 7f0304cee640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:37.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.483+0000 7f0307f54640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:37.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.483+0000 7f0304cee640 1 -- 172.21.15.86:0/1969903185 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f02fc001c00 con 0x7f03000b4700 2024-03-25T18:09:37.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.483+0000 7f0304cee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:37.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.483+0000 7f0307f54640 4 set_mon_vals no callback set 2024-03-25T18:09:37.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.483+0000 7f0304cee640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:37.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.483+0000 7f0304cee640 10 monclient: dump: 2024-03-25T18:09:37.486 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:37.486 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.487 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.487 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.487 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:37.487 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.487 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:37.487 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:37.487 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:37.487 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.484+0000 7f0307f54640 10 monclient: shutdown 2024-03-25T18:09:37.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.484+0000 7f0307f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:37.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.484+0000 7f0307f54640 1 -- 172.21.15.86:0/1969903185 >> v1:172.21.15.86:6789/0 conn(0x7f03000b4700 legacy=0x7f0300142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.484+0000 7f0307f54640 1 -- 172.21.15.86:0/1969903185 shutdown_connections 2024-03-25T18:09:37.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.484+0000 7f0307f54640 1 -- 172.21.15.86:0/1969903185 >> 172.21.15.86:0/1969903185 conn(0x7f03000668e0 msgr2=0x7f0300066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:37.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.484+0000 7f0307f54640 1 -- 172.21.15.86:0/1969903185 shutdown_connections 2024-03-25T18:09:37.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.484+0000 7f0307f54640 1 -- 172.21.15.86:0/1969903185 wait complete. 2024-03-25T18:09:37.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.485+0000 7f0307f54640 10 monclient: build_initial_monmap 2024-03-25T18:09:37.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.485+0000 7f0307f54640 10 monclient: monmap: 2024-03-25T18:09:37.488 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-25T18:09:37.488 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:37.488 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-03-25T18:09:37.488 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-03-25T18:09:37.488 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:37.488 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-25T18:09:37.488 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:37.488 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:09:37.488 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:09:37.488 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.485+0000 7f0307f54640 1 librados: starting msgr at 2024-03-25T18:09:37.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.485+0000 7f0307f54640 1 librados: starting objecter 2024-03-25T18:09:37.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.485+0000 7f0307f54640 1 Processor -- start 2024-03-25T18:09:37.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.485+0000 7f0307f54640 1 -- start start 2024-03-25T18:09:37.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.485+0000 7f0307f54640 1 librados: setting wanted keys 2024-03-25T18:09:37.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.485+0000 7f0307f54640 1 librados: calling monclient init 2024-03-25T18:09:37.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.485+0000 7f0307f54640 10 monclient: init 2024-03-25T18:09:37.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.486+0000 7f0307f54640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:37.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.486+0000 7f0307f54640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:09:37.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.486+0000 7f0307f54640 10 monclient(hunting): picked mon.noname-c con 0x7f03000a96b0 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:37.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.486+0000 7f0307f54640 10 monclient(hunting): picked mon.noname-b con 0x7f03000b4700 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:37.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.486+0000 7f0307f54640 10 monclient(hunting): picked mon.noname-a con 0x7f0300142a20 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:37.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.486+0000 7f0307f54640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03000addd0 con 0x7f0300142a20 2024-03-25T18:09:37.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.486+0000 7f0307f54640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03000ae040 con 0x7f03000b4700 2024-03-25T18:09:37.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.486+0000 7f0307f54640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03000ae320 con 0x7f03000a96b0 2024-03-25T18:09:37.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.486+0000 7f0307f54640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:37.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.486+0000 7f0307f54640 10 monclient(hunting): authenticate will time out at 6958.723102s 2024-03-25T18:09:37.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.486+0000 7f03054ef640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f03000b4700 0x7f030013aa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46422/0 (socket says 172.21.15.86:46422) 2024-03-25T18:09:37.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.486+0000 7f03064f1640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f0300142a20 0x7f03000aca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.86:42696/0 (socket says 172.21.15.86:42696) 2024-03-25T18:09:37.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.486+0000 7f03054ef640 1 -- 172.21.15.86:0/2612355097 learned_addr learned my addr 172.21.15.86:0/2612355097 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-25T18:09:37.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.486+0000 7f02f67fc640 10 client.?.objecter ms_handle_connect 0x7f03000b4700 2024-03-25T18:09:37.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.486+0000 7f02f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:37.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.486+0000 7f02f67fc640 10 client.?.objecter ms_handle_connect 0x7f0300142a20 2024-03-25T18:09:37.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.487+0000 7f02f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:37.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.487+0000 7f02f67fc640 10 client.?.objecter ms_handle_connect 0x7f03000a96b0 2024-03-25T18:09:37.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.487+0000 7f02f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:37.491 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:37.488+0000 7f7080038040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:09:37.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.488+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373161635 0 0) 0x7f03000ae040 con 0x7f03000b4700 2024-03-25T18:09:37.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.488+0000 7f02f67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:37.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.488+0000 7f02f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:37.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.488+0000 7f02f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:37.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.488+0000 7f02f67fc640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:09:37.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.488+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e4004eb0 con 0x7f03000b4700 2024-03-25T18:09:37.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.489+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311936024 0 0) 0x7f03000addd0 con 0x7f0300142a20 2024-03-25T18:09:37.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.489+0000 7f02f67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:37.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.489+0000 7f02f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:37.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.489+0000 7f02f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:37.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.489+0000 7f02f67fc640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:09:37.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.489+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03000ae040 con 0x7f0300142a20 2024-03-25T18:09:37.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.489+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465097208 0 0) 0x7f03000ae320 con 0x7f03000a96b0 2024-03-25T18:09:37.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.489+0000 7f02f67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:37.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.489+0000 7f02f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:37.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.489+0000 7f02f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:37.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.489+0000 7f02f67fc640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:09:37.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.489+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03000addd0 con 0x7f03000a96b0 2024-03-25T18:09:37.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.493+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2090382489 0 0) 0x7f02e4004eb0 con 0x7f03000b4700 2024-03-25T18:09:37.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.493+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03000ae320 con 0x7f03000b4700 2024-03-25T18:09:37.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.493+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f02f0001a10 con 0x7f03000b4700 2024-03-25T18:09:37.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.493+0000 7f02f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:37.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.493+0000 7f02f67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:37.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.493+0000 7f02f67fc640 10 monclient(hunting): dump: 2024-03-25T18:09:37.495 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:37.495 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.495 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.495 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.495 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:37.495 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.495 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:37.495 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:37.495 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:37.495 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.493+0000 7f02f67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:37.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.493+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2321912852 0 0) 0x7f03000ae040 con 0x7f0300142a20 2024-03-25T18:09:37.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.493+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02e4004eb0 con 0x7f0300142a20 2024-03-25T18:09:37.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.493+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f02f8001b80 con 0x7f0300142a20 2024-03-25T18:09:37.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.493+0000 7f02f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:37.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.493+0000 7f02f67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:37.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.493+0000 7f02f67fc640 10 monclient(hunting): dump: 2024-03-25T18:09:37.496 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:37.496 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.496 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.496 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.496 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:37.497 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.497 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:37.497 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:37.497 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:37.497 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.494+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2873867346 0 0) 0x7f03000addd0 con 0x7f03000a96b0 2024-03-25T18:09:37.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.494+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03000ae040 con 0x7f03000a96b0 2024-03-25T18:09:37.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.494+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f02fc001b00 con 0x7f03000a96b0 2024-03-25T18:09:37.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.494+0000 7f02f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:37.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.494+0000 7f02f67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:37.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.494+0000 7f02f67fc640 10 monclient(hunting): dump: 2024-03-25T18:09:37.499 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:37.499 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.499 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.500 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.500 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:37.500 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.500 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:37.500 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:37.500 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:37.500 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.496+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 190896021 0 0) 0x7f03000ae320 con 0x7f03000b4700 2024-03-25T18:09:37.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.496+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 >> v1:172.21.15.64:6790/0 conn(0x7f03000a96b0 legacy=0x7f03000ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.496+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 >> v1:172.21.15.64:6789/0 conn(0x7f0300142a20 legacy=0x7f03000aca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.496+0000 7f02f67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:09:37.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.496+0000 7f02f67fc640 1 monclient: found mon.b 2024-03-25T18:09:37.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.496+0000 7f02f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:37.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.496+0000 7f02f67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:37.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.496+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03000abdd0 con 0x7f03000b4700 2024-03-25T18:09:37.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.496+0000 7f02f67fc640 10 monclient: _finish_auth 0 2024-03-25T18:09:37.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.496+0000 7f02f67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:09:37.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.496+0000 7f02f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:37.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.496+0000 7f0307f54640 5 monclient: authenticate success, global_id 4138 2024-03-25T18:09:37.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.496+0000 7f0307f54640 10 monclient: _renew_subs 2024-03-25T18:09:37.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.496+0000 7f0307f54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:37.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.496+0000 7f0307f54640 1 -- 172.21.15.86:0/2612355097 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03000ad170 con 0x7f03000b4700 2024-03-25T18:09:37.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.497+0000 7f0307f54640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:37.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.497+0000 7f0307f54640 10 monclient: _renew_subs 2024-03-25T18:09:37.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.497+0000 7f0307f54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:37.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.497+0000 7f0307f54640 1 -- 172.21.15.86:0/2612355097 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03001e3480 con 0x7f03000b4700 2024-03-25T18:09:37.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.497+0000 7f0307f54640 1 librados: init done 2024-03-25T18:09:37.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.498+0000 7f0307f54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:37.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.498+0000 7f0307f54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:37.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.498+0000 7f0307f54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:37.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.498+0000 7f0307f54640 1 -- 172.21.15.86:0/2612355097 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03000abfc0 con 0x7f03000b4700 2024-03-25T18:09:37.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.500+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02f0001c70 con 0x7f03000b4700 2024-03-25T18:09:37.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.500+0000 7f02f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:37.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.500+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f02f0003a60 con 0x7f03000b4700 2024-03-25T18:09:37.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.500+0000 7f02f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:37.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.500+0000 7f02f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:37.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.500+0000 7f02f67fc640 10 monclient: dump: 2024-03-25T18:09:37.503 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:37.503 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.504 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.504 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.504 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:37.504 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.504 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:37.504 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:37.504 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:37.504 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.500+0000 7f02f7fff640 4 set_mon_vals no callback set 2024-03-25T18:09:37.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.500+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f02f0004010 con 0x7f03000b4700 2024-03-25T18:09:37.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.500+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 4064638195 0 0) 0x7f02f0006750 con 0x7f03000b4700 2024-03-25T18:09:37.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.500+0000 7f02f67fc640 10 client.4138.objecter ms_dispatch 0x7f0300000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:09:37.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.500+0000 7f02f67fc640 3 client.4138.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:09:37.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.500+0000 7f02f67fc640 3 client.4138.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:09:37.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.500+0000 7f02f67fc640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-25T18:09:37.521 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:37.521+0000 7fb779810040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:09:37.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.594+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f02f00042c0 con 0x7f03000b4700 2024-03-25T18:09:37.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.594+0000 7f02f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:37.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.594+0000 7f02f67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:37.614 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:37.613+0000 7f7080038040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:09:37.620 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:37.620+0000 7fb779810040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:09:37.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.686+0000 7f0307f54640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "315a607b-7263-41dc-8587-bd31883b50d6", "id": 3}] 2024-03-25T18:09:37.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.686+0000 7f0307f54640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "315a607b-7263-41dc-8587-bd31883b50d6", "id": 3}] 2024-03-25T18:09:37.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.686+0000 7f0307f54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:37.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.686+0000 7f0307f54640 1 -- 172.21.15.86:0/2612355097 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd new", "uuid": "315a607b-7263-41dc-8587-bd31883b50d6", "id": 3} v 0) v1 -- 0x7f030004f730 con 0x7f03000b4700 2024-03-25T18:09:37.720 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:37.719+0000 7f7080038040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:09:37.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.751+0000 7f02f67fc640 1 -- 172.21.15.86:0/2612355097 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "315a607b-7263-41dc-8587-bd31883b50d6", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 399696075 0 525046306) 0x7f02f0006370 con 0x7f03000b4700 2024-03-25T18:09:37.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.752+0000 7f02f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "315a607b-7263-41dc-8587-bd31883b50d6", "id": 3}] 2024-03-25T18:09:37.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.752+0000 7f02f67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:37.753 INFO:teuthology.orchestra.run.smithi086.stdout:3 2024-03-25T18:09:37.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.752+0000 7f0307f54640 10 librados: watch_flush enter 2024-03-25T18:09:37.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.752+0000 7f0307f54640 10 librados: watch_flush exit 2024-03-25T18:09:37.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.752+0000 7f0307f54640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-25T18:09:37.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.752+0000 7f0307f54640 10 client.4138.objecter successfully canceled tick 2024-03-25T18:09:37.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.753+0000 7f0307f54640 10 monclient: shutdown 2024-03-25T18:09:37.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.753+0000 7f0307f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:37.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.753+0000 7f0307f54640 1 -- 172.21.15.86:0/2612355097 >> v1:172.21.15.86:6789/0 conn(0x7f03000b4700 legacy=0x7f030013aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.753+0000 7f0307f54640 1 -- 172.21.15.86:0/2612355097 shutdown_connections 2024-03-25T18:09:37.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.753+0000 7f0307f54640 1 -- 172.21.15.86:0/2612355097 >> 172.21.15.86:0/2612355097 conn(0x7f03000668e0 msgr2=0x7f03000b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:37.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.753+0000 7f0307f54640 1 -- 172.21.15.86:0/2612355097 shutdown_connections 2024-03-25T18:09:37.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.753+0000 7f0307f54640 1 -- 172.21.15.86:0/2612355097 wait complete. 2024-03-25T18:09:37.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.753+0000 7f0307f54640 1 librados: shutdown 2024-03-25T18:09:37.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 95fe487e-5eef-4187-86b8-182e953b95c7 4 2024-03-25T18:09:37.790 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:37.789+0000 7fb779810040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:09:37.836 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:37.835+0000 7fb779810040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:09:37.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.865+0000 7f1040dc4640 10 monclient: get_monmap_and_config 2024-03-25T18:09:37.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.865+0000 7f1040dc4640 10 monclient: build_initial_monmap 2024-03-25T18:09:37.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.865+0000 7f1040dc4640 10 monclient: monmap: 2024-03-25T18:09:37.866 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-25T18:09:37.866 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.866 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:09:37.866601+0000 2024-03-25T18:09:37.866 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:09:37.866601+0000 2024-03-25T18:09:37.866 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:37.866 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.866 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:37.867 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:09:37.867 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:09:37.867 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.866+0000 7f1040dc4640 1 Processor -- start 2024-03-25T18:09:37.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.866+0000 7f1040dc4640 1 -- start start 2024-03-25T18:09:37.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.866+0000 7f1040dc4640 10 monclient: init 2024-03-25T18:09:37.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.866+0000 7f1040dc4640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:37.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.866+0000 7f1040dc4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:37.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.866+0000 7f1040dc4640 10 monclient(hunting): picked mon.noname-c con 0x7f103c0b4700 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:37.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.866+0000 7f1040dc4640 10 monclient(hunting): picked mon.noname-b con 0x7f103c142a20 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:37.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.866+0000 7f1040dc4640 10 monclient(hunting): picked mon.noname-a con 0x7f103c0a96b0 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:37.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.866+0000 7f1040dc4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103c0a9d60 con 0x7f103c0a96b0 2024-03-25T18:09:37.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.866+0000 7f1040dc4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103c13ff10 con 0x7f103c0b4700 2024-03-25T18:09:37.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.866+0000 7f1040dc4640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103c0aa1f0 con 0x7f103c142a20 2024-03-25T18:09:37.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.866+0000 7f1040dc4640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:37.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.866+0000 7f1040dc4640 10 monclient(hunting): authenticate will time out at 6959.103801s 2024-03-25T18:09:37.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.867+0000 7f103a59c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f103c0b4700 0x7f103c142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46434/0 (socket says 172.21.15.86:46434) 2024-03-25T18:09:37.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.867+0000 7f103a59c640 1 -- 172.21.15.86:0/2123781548 learned_addr learned my addr 172.21.15.86:0/2123781548 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-25T18:09:37.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.869+0000 7f1039d9b640 1 -- 172.21.15.86:0/2123781548 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952047801 0 0) 0x7f103c13ff10 con 0x7f103c0b4700 2024-03-25T18:09:37.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.869+0000 7f1039d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:37.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.869+0000 7f1039d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:37.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.869+0000 7f1039d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:37.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.869+0000 7f1039d9b640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:09:37.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.869+0000 7f1039d9b640 1 -- 172.21.15.86:0/2123781548 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1028003520 con 0x7f103c0b4700 2024-03-25T18:09:37.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.869+0000 7f1039d9b640 1 -- 172.21.15.86:0/2123781548 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754542555 0 0) 0x7f103c0a9d60 con 0x7f103c0a96b0 2024-03-25T18:09:37.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.869+0000 7f1039d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:37.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.869+0000 7f1039d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:37.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.869+0000 7f1039d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:37.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.869+0000 7f1039d9b640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:09:37.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.869+0000 7f1039d9b640 1 -- 172.21.15.86:0/2123781548 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f103c13ff10 con 0x7f103c0a96b0 2024-03-25T18:09:37.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.870+0000 7f1039d9b640 1 -- 172.21.15.86:0/2123781548 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2078134931 0 0) 0x7f103c0aa1f0 con 0x7f103c142a20 2024-03-25T18:09:37.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.870+0000 7f1039d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:37.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.870+0000 7f1039d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:37.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.870+0000 7f1039d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:37.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.870+0000 7f1039d9b640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:09:37.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.870+0000 7f1039d9b640 1 -- 172.21.15.86:0/2123781548 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f103c0a9d60 con 0x7f103c142a20 2024-03-25T18:09:37.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.872+0000 7f1039d9b640 1 -- 172.21.15.86:0/2123781548 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1977071927 0 0) 0x7f1028003520 con 0x7f103c0b4700 2024-03-25T18:09:37.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.872+0000 7f1039d9b640 1 -- 172.21.15.86:0/2123781548 >> v1:172.21.15.64:6790/0 conn(0x7f103c142a20 legacy=0x7f103c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.872+0000 7f1039d9b640 1 -- 172.21.15.86:0/2123781548 >> v1:172.21.15.64:6789/0 conn(0x7f103c0a96b0 legacy=0x7f103c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.872+0000 7f1039d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:09:37.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.872+0000 7f1039d9b640 1 monclient: found mon.noname-c 2024-03-25T18:09:37.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.873+0000 7f1039d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:37.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.873+0000 7f1039d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:09:37.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.873+0000 7f1039d9b640 1 -- 172.21.15.86:0/2123781548 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103c0b4ae0 con 0x7f103c0b4700 2024-03-25T18:09:37.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.873+0000 7f1039d9b640 10 monclient: _finish_auth 0 2024-03-25T18:09:37.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.873+0000 7f1039d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:09:37.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.873+0000 7f1039d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:37.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.873+0000 7f1039d9b640 1 -- 172.21.15.86:0/2123781548 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f102c001690 con 0x7f103c0b4700 2024-03-25T18:09:37.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.873+0000 7f1040dc4640 5 monclient: authenticate success, global_id 4141 2024-03-25T18:09:37.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.873+0000 7f1040dc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:37.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.873+0000 7f1039d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:37.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.873+0000 7f1039d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:37.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.873+0000 7f1039d9b640 10 monclient: dump: 2024-03-25T18:09:37.875 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:37.875 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.875 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.875 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.875 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:37.875 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.875 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:37.875 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:37.875 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:37.875 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.873+0000 7f1040dc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:37.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.874+0000 7f1039d9b640 1 -- 172.21.15.86:0/2123781548 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f102c0017f0 con 0x7f103c0b4700 2024-03-25T18:09:37.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.874+0000 7f1039d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:37.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.874+0000 7f1039d9b640 1 -- 172.21.15.86:0/2123781548 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f102c001c00 con 0x7f103c0b4700 2024-03-25T18:09:37.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.874+0000 7f1040dc4640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:37.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.874+0000 7f1040dc4640 4 set_mon_vals no callback set 2024-03-25T18:09:37.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.874+0000 7f1039d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:37.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.874+0000 7f1039d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:37.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.874+0000 7f1039d9b640 10 monclient: dump: 2024-03-25T18:09:37.877 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:37.877 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.877 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:37.878 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.875+0000 7f1040dc4640 10 monclient: shutdown 2024-03-25T18:09:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.875+0000 7f1040dc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.875+0000 7f1040dc4640 1 -- 172.21.15.86:0/2123781548 >> v1:172.21.15.86:6789/0 conn(0x7f103c0b4700 legacy=0x7f103c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.875+0000 7f1040dc4640 1 -- 172.21.15.86:0/2123781548 shutdown_connections 2024-03-25T18:09:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.875+0000 7f1040dc4640 1 -- 172.21.15.86:0/2123781548 >> 172.21.15.86:0/2123781548 conn(0x7f103c0668e0 msgr2=0x7f103c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.875+0000 7f1040dc4640 1 -- 172.21.15.86:0/2123781548 shutdown_connections 2024-03-25T18:09:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.875+0000 7f1040dc4640 1 -- 172.21.15.86:0/2123781548 wait complete. 2024-03-25T18:09:37.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.876+0000 7f1040dc4640 10 monclient: build_initial_monmap 2024-03-25T18:09:37.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.876+0000 7f1040dc4640 10 monclient: monmap: 2024-03-25T18:09:37.879 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-25T18:09:37.879 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:37.879 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-03-25T18:09:37.879 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-03-25T18:09:37.879 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:37.879 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-25T18:09:37.879 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:37.879 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:09:37.879 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:09:37.879 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.876+0000 7f1040dc4640 1 librados: starting msgr at 2024-03-25T18:09:37.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.876+0000 7f1040dc4640 1 librados: starting objecter 2024-03-25T18:09:37.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.876+0000 7f1040dc4640 1 Processor -- start 2024-03-25T18:09:37.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.876+0000 7f1040dc4640 1 -- start start 2024-03-25T18:09:37.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.876+0000 7f1040dc4640 1 librados: setting wanted keys 2024-03-25T18:09:37.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.876+0000 7f1040dc4640 1 librados: calling monclient init 2024-03-25T18:09:37.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.876+0000 7f1040dc4640 10 monclient: init 2024-03-25T18:09:37.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.877+0000 7f1040dc4640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:37.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.877+0000 7f1040dc4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:09:37.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.877+0000 7f1040dc4640 10 monclient(hunting): picked mon.noname-c con 0x7f103c0a96b0 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:37.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.877+0000 7f1040dc4640 10 monclient(hunting): picked mon.noname-a con 0x7f103c0b4700 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:37.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.877+0000 7f1040dc4640 10 monclient(hunting): picked mon.noname-b con 0x7f103c142a20 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:37.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.877+0000 7f1040dc4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103c13a960 con 0x7f103c0b4700 2024-03-25T18:09:37.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.877+0000 7f1040dc4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103c139340 con 0x7f103c142a20 2024-03-25T18:09:37.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.877+0000 7f1040dc4640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103c138db0 con 0x7f103c0a96b0 2024-03-25T18:09:37.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.877+0000 7f1040dc4640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:37.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.877+0000 7f1040dc4640 10 monclient(hunting): authenticate will time out at 6959.114311s 2024-03-25T18:09:37.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.877+0000 7f103ad9d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f103c142a20 0x7f103c1daf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46440/0 (socket says 172.21.15.86:46440) 2024-03-25T18:09:37.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.877+0000 7f103ad9d640 1 -- 172.21.15.86:0/2775307372 learned_addr learned my addr 172.21.15.86:0/2775307372 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-25T18:09:37.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.878+0000 7f1032ffd640 10 client.?.objecter ms_handle_connect 0x7f103c142a20 2024-03-25T18:09:37.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.878+0000 7f1032ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:37.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.878+0000 7f1032ffd640 10 client.?.objecter ms_handle_connect 0x7f103c0b4700 2024-03-25T18:09:37.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.878+0000 7f1032ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:37.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.878+0000 7f1032ffd640 10 client.?.objecter ms_handle_connect 0x7f103c0a96b0 2024-03-25T18:09:37.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.878+0000 7f1032ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:37.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.880+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493725023 0 0) 0x7f103c139340 con 0x7f103c142a20 2024-03-25T18:09:37.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.880+0000 7f1032ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:37.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.880+0000 7f1032ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:37.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.880+0000 7f1032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:37.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.880+0000 7f1032ffd640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:09:37.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.880+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f101c004eb0 con 0x7f103c142a20 2024-03-25T18:09:37.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.880+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083726736 0 0) 0x7f103c13a960 con 0x7f103c0b4700 2024-03-25T18:09:37.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.880+0000 7f1032ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:37.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.880+0000 7f1032ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:37.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.880+0000 7f1032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:37.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.880+0000 7f1032ffd640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:09:37.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.880+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f103c139340 con 0x7f103c0b4700 2024-03-25T18:09:37.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.880+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167712028 0 0) 0x7f103c138db0 con 0x7f103c0a96b0 2024-03-25T18:09:37.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.880+0000 7f1032ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:37.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.880+0000 7f1032ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:37.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.880+0000 7f1032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:37.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.880+0000 7f1032ffd640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:09:37.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.880+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f103c13a960 con 0x7f103c0a96b0 2024-03-25T18:09:37.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.884+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1425385885 0 0) 0x7f101c004eb0 con 0x7f103c142a20 2024-03-25T18:09:37.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.885+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f103c138db0 con 0x7f103c142a20 2024-03-25T18:09:37.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.885+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1034001d30 con 0x7f103c142a20 2024-03-25T18:09:37.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.885+0000 7f1032ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:37.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.885+0000 7f1032ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:37.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.885+0000 7f1032ffd640 10 monclient(hunting): dump: 2024-03-25T18:09:37.886 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:37.886 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.886 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.886 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.886 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:37.886 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.886 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:37.886 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:37.886 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:37.886 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.885+0000 7f1032ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:37.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.886+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4057554537 0 0) 0x7f103c139340 con 0x7f103c0b4700 2024-03-25T18:09:37.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.886+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f101c004eb0 con 0x7f103c0b4700 2024-03-25T18:09:37.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.886+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f10240017d0 con 0x7f103c0b4700 2024-03-25T18:09:37.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.886+0000 7f1032ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:37.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.886+0000 7f1032ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:37.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.886+0000 7f1032ffd640 10 monclient(hunting): dump: 2024-03-25T18:09:37.887 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:37.887 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.887 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.888 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.888 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:37.888 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.888 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:37.888 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:37.888 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:37.888 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4114884298 0 0) 0x7f103c138db0 con 0x7f103c142a20 2024-03-25T18:09:37.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 >> v1:172.21.15.64:6790/0 conn(0x7f103c0a96b0 legacy=0x7f103c130f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 >> v1:172.21.15.64:6789/0 conn(0x7f103c0b4700 legacy=0x7f103c1da7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1032ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:09:37.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1032ffd640 1 monclient: found mon.b 2024-03-25T18:09:37.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1032ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:37.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1032ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:37.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103c138f40 con 0x7f103c142a20 2024-03-25T18:09:37.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1032ffd640 10 monclient: _finish_auth 0 2024-03-25T18:09:37.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1032ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:09:37.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1032ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:37.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1040dc4640 5 monclient: authenticate success, global_id 4144 2024-03-25T18:09:37.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1040dc4640 10 monclient: _renew_subs 2024-03-25T18:09:37.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1040dc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:37.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1040dc4640 1 -- 172.21.15.86:0/2775307372 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f103c138750 con 0x7f103c142a20 2024-03-25T18:09:37.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1040dc4640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:37.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1040dc4640 10 monclient: _renew_subs 2024-03-25T18:09:37.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1040dc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:37.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1040dc4640 1 -- 172.21.15.86:0/2775307372 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f103c131820 con 0x7f103c142a20 2024-03-25T18:09:37.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.887+0000 7f1040dc4640 1 librados: init done 2024-03-25T18:09:37.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.888+0000 7f1040dc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:37.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.888+0000 7f1040dc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:37.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.888+0000 7f1040dc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:37.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.888+0000 7f1040dc4640 1 -- 172.21.15.86:0/2775307372 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f103c139190 con 0x7f103c142a20 2024-03-25T18:09:37.899 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:37.898+0000 7f7080038040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:09:37.942 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:37.942+0000 7f7080038040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:09:37.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.991+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1034003990 con 0x7f103c142a20 2024-03-25T18:09:37.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.991+0000 7f1032ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:37.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.991+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1034003d20 con 0x7f103c142a20 2024-03-25T18:09:37.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.991+0000 7f1032ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:37.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.991+0000 7f1032ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:37.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.991+0000 7f1032ffd640 10 monclient: dump: 2024-03-25T18:09:37.992 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:37.992 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:37.992 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.992 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:37.993 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:37.993 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:37.993 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:37.993 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:37.993 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:37.993 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:37.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.991+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f10340042d0 con 0x7f103c142a20 2024-03-25T18:09:37.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.991+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 159826241 0 0) 0x7f1034006a10 con 0x7f103c142a20 2024-03-25T18:09:37.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.991+0000 7f1032ffd640 10 client.4144.objecter ms_dispatch 0x7f103c000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:09:37.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.991+0000 7f103959a640 4 set_mon_vals no callback set 2024-03-25T18:09:37.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.991+0000 7f1032ffd640 3 client.4144.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:09:37.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.991+0000 7f1032ffd640 3 client.4144.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:09:37.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.991+0000 7f1032ffd640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T18:09:37.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.991+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1034018e00 con 0x7f103c142a20 2024-03-25T18:09:37.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.991+0000 7f1032ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:37.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:37.991+0000 7f1032ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:38.002 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:38.001+0000 7fb779810040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:09:38.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.064+0000 7f1040dc4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "95fe487e-5eef-4187-86b8-182e953b95c7", "id": 4}] 2024-03-25T18:09:38.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.064+0000 7f1040dc4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "95fe487e-5eef-4187-86b8-182e953b95c7", "id": 4}] 2024-03-25T18:09:38.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.064+0000 7f1040dc4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:38.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.064+0000 7f1040dc4640 1 -- 172.21.15.86:0/2775307372 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd new", "uuid": "95fe487e-5eef-4187-86b8-182e953b95c7", "id": 4} v 0) v1 -- 0x7f103c138af0 con 0x7f103c142a20 2024-03-25T18:09:38.101 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:38.101+0000 7f7080038040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:09:38.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.142+0000 7f1032ffd640 1 -- 172.21.15.86:0/2775307372 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "95fe487e-5eef-4187-86b8-182e953b95c7", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 4049047390 0 1697012583) 0x7f1034004650 con 0x7f103c142a20 2024-03-25T18:09:38.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.142+0000 7f1032ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "95fe487e-5eef-4187-86b8-182e953b95c7", "id": 4}] 2024-03-25T18:09:38.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.142+0000 7f1032ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:38.143 INFO:teuthology.orchestra.run.smithi086.stdout:4 2024-03-25T18:09:38.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.143+0000 7f1040dc4640 10 librados: watch_flush enter 2024-03-25T18:09:38.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.143+0000 7f1040dc4640 10 librados: watch_flush exit 2024-03-25T18:09:38.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.143+0000 7f1040dc4640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T18:09:38.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.143+0000 7f1040dc4640 10 client.4144.objecter successfully canceled tick 2024-03-25T18:09:38.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.143+0000 7f1040dc4640 10 monclient: shutdown 2024-03-25T18:09:38.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.143+0000 7f1040dc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:38.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.143+0000 7f1040dc4640 1 -- 172.21.15.86:0/2775307372 >> v1:172.21.15.86:6789/0 conn(0x7f103c142a20 legacy=0x7f103c1daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:38.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.143+0000 7f1040dc4640 1 -- 172.21.15.86:0/2775307372 shutdown_connections 2024-03-25T18:09:38.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.143+0000 7f1040dc4640 1 -- 172.21.15.86:0/2775307372 >> 172.21.15.86:0/2775307372 conn(0x7f103c0668e0 msgr2=0x7f103c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:38.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.143+0000 7f1040dc4640 1 -- 172.21.15.86:0/2775307372 shutdown_connections 2024-03-25T18:09:38.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.143+0000 7f1040dc4640 1 -- 172.21.15.86:0/2775307372 wait complete. 2024-03-25T18:09:38.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.143+0000 7f1040dc4640 1 librados: shutdown 2024-03-25T18:09:38.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new b99f705f-8c5b-4b56-9e68-e6308b38ac09 5 2024-03-25T18:09:38.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.254+0000 7fbf94ff6640 10 monclient: get_monmap_and_config 2024-03-25T18:09:38.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.254+0000 7fbf94ff6640 10 monclient: build_initial_monmap 2024-03-25T18:09:38.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.254+0000 7fbf94ff6640 10 monclient: monmap: 2024-03-25T18:09:38.255 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-25T18:09:38.256 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:38.256 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:09:38.255967+0000 2024-03-25T18:09:38.256 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:09:38.255967+0000 2024-03-25T18:09:38.256 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:38.256 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:38.256 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:38.256 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:09:38.256 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:09:38.256 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:38.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.255+0000 7fbf94ff6640 1 Processor -- start 2024-03-25T18:09:38.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.255+0000 7fbf94ff6640 1 -- start start 2024-03-25T18:09:38.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.255+0000 7fbf94ff6640 10 monclient: init 2024-03-25T18:09:38.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.256+0000 7fbf94ff6640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:38.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.256+0000 7fbf94ff6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:38.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.256+0000 7fbf94ff6640 10 monclient(hunting): picked mon.noname-a con 0x7fbf900b1d40 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:38.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.256+0000 7fbf94ff6640 10 monclient(hunting): picked mon.noname-b con 0x7fbf900b14e0 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:38.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.256+0000 7fbf94ff6640 10 monclient(hunting): picked mon.noname-c con 0x7fbf900b4400 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:38.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.256+0000 7fbf94ff6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9013ff90 con 0x7fbf900b1d40 2024-03-25T18:09:38.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.256+0000 7fbf94ff6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf900a9a60 con 0x7fbf900b4400 2024-03-25T18:09:38.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.256+0000 7fbf94ff6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf900ab4a0 con 0x7fbf900b14e0 2024-03-25T18:09:38.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.256+0000 7fbf94ff6640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:38.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.256+0000 7fbf94ff6640 10 monclient(hunting): authenticate will time out at 6959.493237s 2024-03-25T18:09:38.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.256+0000 7fbf8e59c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fbf900b1d40 0x7fbf900b0dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.86:42728/0 (socket says 172.21.15.86:42728) 2024-03-25T18:09:38.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.256+0000 7fbf8e59c640 1 -- 172.21.15.86:0/2096176877 learned_addr learned my addr 172.21.15.86:0/2096176877 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-25T18:09:38.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.259+0000 7fbf8d59a640 1 -- 172.21.15.86:0/2096176877 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793889403 0 0) 0x7fbf9013ff90 con 0x7fbf900b1d40 2024-03-25T18:09:38.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.259+0000 7fbf8d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:09:38.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.259+0000 7fbf8d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:38.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.259+0000 7fbf8d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:38.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.259+0000 7fbf8d59a640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:09:38.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.259+0000 7fbf8d59a640 1 -- 172.21.15.86:0/2096176877 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf78003520 con 0x7fbf900b1d40 2024-03-25T18:09:38.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.259+0000 7fbf8d59a640 1 -- 172.21.15.86:0/2096176877 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063342303 0 0) 0x7fbf900a9a60 con 0x7fbf900b4400 2024-03-25T18:09:38.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.259+0000 7fbf8d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:09:38.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.259+0000 7fbf8d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:38.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.259+0000 7fbf8d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:38.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.259+0000 7fbf8d59a640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:09:38.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.259+0000 7fbf8d59a640 1 -- 172.21.15.86:0/2096176877 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf9013ff90 con 0x7fbf900b4400 2024-03-25T18:09:38.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.259+0000 7fbf8d59a640 1 -- 172.21.15.86:0/2096176877 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341592681 0 0) 0x7fbf900ab4a0 con 0x7fbf900b14e0 2024-03-25T18:09:38.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.259+0000 7fbf8d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:09:38.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.259+0000 7fbf8d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:38.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.259+0000 7fbf8d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:38.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.259+0000 7fbf8d59a640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:09:38.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.259+0000 7fbf8d59a640 1 -- 172.21.15.86:0/2096176877 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf900a9a60 con 0x7fbf900b14e0 2024-03-25T18:09:38.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.262+0000 7fbf8d59a640 1 -- 172.21.15.86:0/2096176877 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3755929222 0 0) 0x7fbf78003520 con 0x7fbf900b1d40 2024-03-25T18:09:38.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.262+0000 7fbf8d59a640 1 -- 172.21.15.86:0/2096176877 >> v1:172.21.15.64:6790/0 conn(0x7fbf900b14e0 legacy=0x7fbf90142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:38.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.262+0000 7fbf8d59a640 1 -- 172.21.15.86:0/2096176877 >> v1:172.21.15.86:6789/0 conn(0x7fbf900b4400 legacy=0x7fbf9014dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:38.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.262+0000 7fbf8d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:09:38.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.262+0000 7fbf8d59a640 1 monclient: found mon.noname-a 2024-03-25T18:09:38.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.262+0000 7fbf8d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:38.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.262+0000 7fbf8d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:09:38.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.262+0000 7fbf8d59a640 1 -- 172.21.15.86:0/2096176877 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf900a9540 con 0x7fbf900b1d40 2024-03-25T18:09:38.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.262+0000 7fbf8d59a640 10 monclient: _finish_auth 0 2024-03-25T18:09:38.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.262+0000 7fbf8d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:09:38.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.262+0000 7fbf8d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:38.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.262+0000 7fbf8d59a640 1 -- 172.21.15.86:0/2096176877 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbf84001690 con 0x7fbf900b1d40 2024-03-25T18:09:38.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.262+0000 7fbf8d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:38.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.262+0000 7fbf8d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:38.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.262+0000 7fbf8d59a640 10 monclient: dump: 2024-03-25T18:09:38.264 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:38.265 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:38.265 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.265 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.265 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:38.265 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:38.265 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:38.265 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:38.265 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:38.265 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:38.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.262+0000 7fbf94ff6640 5 monclient: authenticate success, global_id 4146 2024-03-25T18:09:38.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.263+0000 7fbf94ff6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:38.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.264+0000 7fbf8d59a640 1 -- 172.21.15.86:0/2096176877 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf840017f0 con 0x7fbf900b1d40 2024-03-25T18:09:38.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.264+0000 7fbf8d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:38.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.264+0000 7fbf8d59a640 1 -- 172.21.15.86:0/2096176877 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbf84001c00 con 0x7fbf900b1d40 2024-03-25T18:09:38.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.264+0000 7fbf94ff6640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:38.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.264+0000 7fbf94ff6640 4 set_mon_vals no callback set 2024-03-25T18:09:38.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.264+0000 7fbf8d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:38.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.264+0000 7fbf8d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:38.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.264+0000 7fbf8d59a640 10 monclient: dump: 2024-03-25T18:09:38.267 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:38.267 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:38.267 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.267 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.267 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:38.267 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:38.267 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:38.267 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:38.267 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:38.267 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:38.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.265+0000 7fbf94ff6640 10 monclient: shutdown 2024-03-25T18:09:38.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.265+0000 7fbf94ff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:38.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.265+0000 7fbf94ff6640 1 -- 172.21.15.86:0/2096176877 >> v1:172.21.15.64:6789/0 conn(0x7fbf900b1d40 legacy=0x7fbf900b0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:38.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.265+0000 7fbf94ff6640 1 -- 172.21.15.86:0/2096176877 shutdown_connections 2024-03-25T18:09:38.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.265+0000 7fbf94ff6640 1 -- 172.21.15.86:0/2096176877 >> 172.21.15.86:0/2096176877 conn(0x7fbf900b7df0 msgr2=0x7fbf900b81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:38.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.265+0000 7fbf94ff6640 1 -- 172.21.15.86:0/2096176877 shutdown_connections 2024-03-25T18:09:38.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.265+0000 7fbf94ff6640 1 -- 172.21.15.86:0/2096176877 wait complete. 2024-03-25T18:09:38.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.266+0000 7fbf94ff6640 10 monclient: build_initial_monmap 2024-03-25T18:09:38.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.266+0000 7fbf94ff6640 10 monclient: monmap: 2024-03-25T18:09:38.270 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-25T18:09:38.270 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:38.270 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-03-25T18:09:38.270 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-03-25T18:09:38.270 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:38.270 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-25T18:09:38.270 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:38.271 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:09:38.271 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:09:38.271 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:38.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.266+0000 7fbf94ff6640 1 librados: starting msgr at 2024-03-25T18:09:38.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.266+0000 7fbf94ff6640 1 librados: starting objecter 2024-03-25T18:09:38.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.266+0000 7fbf94ff6640 1 Processor -- start 2024-03-25T18:09:38.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.266+0000 7fbf94ff6640 1 -- start start 2024-03-25T18:09:38.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.266+0000 7fbf94ff6640 1 librados: setting wanted keys 2024-03-25T18:09:38.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.266+0000 7fbf94ff6640 1 librados: calling monclient init 2024-03-25T18:09:38.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.266+0000 7fbf94ff6640 10 monclient: init 2024-03-25T18:09:38.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.267+0000 7fbf94ff6640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:38.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.267+0000 7fbf94ff6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:38.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.267+0000 7fbf94ff6640 10 monclient(hunting): picked mon.noname-b con 0x7fbf900b14e0 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:38.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.267+0000 7fbf94ff6640 10 monclient(hunting): picked mon.noname-c con 0x7fbf900b1d40 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:38.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.267+0000 7fbf94ff6640 10 monclient(hunting): picked mon.noname-a con 0x7fbf900b4400 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:38.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.267+0000 7fbf94ff6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf900af860 con 0x7fbf900b4400 2024-03-25T18:09:38.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.267+0000 7fbf94ff6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf900afb00 con 0x7fbf900b14e0 2024-03-25T18:09:38.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.267+0000 7fbf94ff6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf900b0a50 con 0x7fbf900b1d40 2024-03-25T18:09:38.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.267+0000 7fbf94ff6640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:38.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.267+0000 7fbf94ff6640 10 monclient(hunting): authenticate will time out at 6959.504376s 2024-03-25T18:09:38.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.267+0000 7fbf8e59c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fbf900b14e0 0x7fbf900af150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46460/0 (socket says 172.21.15.86:46460) 2024-03-25T18:09:38.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.267+0000 7fbf8e59c640 1 -- 172.21.15.86:0/2173581792 learned_addr learned my addr 172.21.15.86:0/2173581792 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-25T18:09:38.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.268+0000 7fbf7effd640 10 client.?.objecter ms_handle_connect 0x7fbf900b14e0 2024-03-25T18:09:38.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.268+0000 7fbf7effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:38.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.268+0000 7fbf7effd640 10 client.?.objecter ms_handle_connect 0x7fbf900b4400 2024-03-25T18:09:38.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.268+0000 7fbf7effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:38.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.268+0000 7fbf7effd640 10 client.?.objecter ms_handle_connect 0x7fbf900b1d40 2024-03-25T18:09:38.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.268+0000 7fbf7effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:38.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.270+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848636650 0 0) 0x7fbf900afb00 con 0x7fbf900b14e0 2024-03-25T18:09:38.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.270+0000 7fbf7effd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:38.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.270+0000 7fbf7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:38.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.270+0000 7fbf7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:38.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.270+0000 7fbf7effd640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:09:38.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.270+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf70004eb0 con 0x7fbf900b14e0 2024-03-25T18:09:38.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.270+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874522827 0 0) 0x7fbf900af860 con 0x7fbf900b4400 2024-03-25T18:09:38.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.270+0000 7fbf7effd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:38.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.270+0000 7fbf7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:38.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.270+0000 7fbf7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:38.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.270+0000 7fbf7effd640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:09:38.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.270+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf900afb00 con 0x7fbf900b4400 2024-03-25T18:09:38.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.270+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234830988 0 0) 0x7fbf900b0a50 con 0x7fbf900b1d40 2024-03-25T18:09:38.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.270+0000 7fbf7effd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:38.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.270+0000 7fbf7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:38.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.270+0000 7fbf7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:38.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.270+0000 7fbf7effd640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:09:38.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.270+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf900af860 con 0x7fbf900b1d40 2024-03-25T18:09:38.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.274+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3577849141 0 0) 0x7fbf900af860 con 0x7fbf900b1d40 2024-03-25T18:09:38.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.274+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf900b0a50 con 0x7fbf900b1d40 2024-03-25T18:09:38.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.274+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbf80001830 con 0x7fbf900b1d40 2024-03-25T18:09:38.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.274+0000 7fbf7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:38.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.274+0000 7fbf7effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:38.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.274+0000 7fbf7effd640 10 monclient(hunting): dump: 2024-03-25T18:09:38.276 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:38.276 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:38.276 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.276 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.276 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:38.277 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:38.277 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:38.277 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:38.277 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:38.277 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:38.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.274+0000 7fbf7effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:38.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.274+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 445753411 0 0) 0x7fbf70004eb0 con 0x7fbf900b14e0 2024-03-25T18:09:38.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.274+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf900af860 con 0x7fbf900b14e0 2024-03-25T18:09:38.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.274+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbf84001bc0 con 0x7fbf900b14e0 2024-03-25T18:09:38.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.274+0000 7fbf7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:38.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.274+0000 7fbf7effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:38.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.274+0000 7fbf7effd640 10 monclient(hunting): dump: 2024-03-25T18:09:38.277 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:38.277 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:38.278 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.278 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.278 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:38.278 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:38.278 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:38.278 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:38.278 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:38.278 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:38.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.275+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3270923974 0 0) 0x7fbf900afb00 con 0x7fbf900b4400 2024-03-25T18:09:38.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.275+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf70004eb0 con 0x7fbf900b4400 2024-03-25T18:09:38.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.275+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbf88001b50 con 0x7fbf900b4400 2024-03-25T18:09:38.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.275+0000 7fbf7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:38.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.275+0000 7fbf7effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:38.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.275+0000 7fbf7effd640 10 monclient(hunting): dump: 2024-03-25T18:09:38.278 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:38.279 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:38.279 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.279 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.279 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:38.279 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:38.279 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:38.279 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:38.279 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:38.279 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:38.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2985579534 0 0) 0x7fbf900af860 con 0x7fbf900b14e0 2024-03-25T18:09:38.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 >> v1:172.21.15.64:6790/0 conn(0x7fbf900b1d40 legacy=0x7fbf9013df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:38.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 >> v1:172.21.15.64:6789/0 conn(0x7fbf900b4400 legacy=0x7fbf901d73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:38.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf7effd640 10 monclient: _finish_hunting 0 2024-03-25T18:09:38.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf7effd640 1 monclient: found mon.b 2024-03-25T18:09:38.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:38.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf7effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:38.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9013e6a0 con 0x7fbf900b14e0 2024-03-25T18:09:38.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf7effd640 10 monclient: _finish_auth 0 2024-03-25T18:09:38.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf7effd640 10 monclient: _check_auth_tickets 2024-03-25T18:09:38.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:38.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf94ff6640 5 monclient: authenticate success, global_id 4150 2024-03-25T18:09:38.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf94ff6640 10 monclient: _renew_subs 2024-03-25T18:09:38.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf94ff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:38.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf94ff6640 1 -- 172.21.15.86:0/2173581792 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9013e8f0 con 0x7fbf900b14e0 2024-03-25T18:09:38.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf94ff6640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:38.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf94ff6640 10 monclient: _renew_subs 2024-03-25T18:09:38.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf94ff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:38.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf94ff6640 1 -- 172.21.15.86:0/2173581792 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf900b0120 con 0x7fbf900b14e0 2024-03-25T18:09:38.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.278+0000 7fbf94ff6640 1 librados: init done 2024-03-25T18:09:38.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.279+0000 7fbf94ff6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:38.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.279+0000 7fbf94ff6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:38.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.279+0000 7fbf94ff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:38.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.279+0000 7fbf94ff6640 1 -- 172.21.15.86:0/2173581792 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf900afd40 con 0x7fbf900b14e0 2024-03-25T18:09:38.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.379+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf840011f0 con 0x7fbf900b14e0 2024-03-25T18:09:38.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.379+0000 7fbf7effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:38.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.379+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbf84004b10 con 0x7fbf900b14e0 2024-03-25T18:09:38.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.379+0000 7fbf7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:38.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.379+0000 7fbf7effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:38.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.379+0000 7fbf7effd640 10 monclient: dump: 2024-03-25T18:09:38.380 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:38.380 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:38.380 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.380 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.380 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:38.381 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:38.381 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:38.381 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:38.381 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:38.381 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:38.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.379+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbf84005080 con 0x7fbf900b14e0 2024-03-25T18:09:38.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.379+0000 7fbf8cd99640 4 set_mon_vals no callback set 2024-03-25T18:09:38.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.379+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2129671735 0 0) 0x7fbf84006820 con 0x7fbf900b14e0 2024-03-25T18:09:38.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.379+0000 7fbf7effd640 10 client.4150.objecter ms_dispatch 0x7fbf90000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:09:38.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.379+0000 7fbf7effd640 3 client.4150.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:09:38.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.379+0000 7fbf7effd640 3 client.4150.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:09:38.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.379+0000 7fbf7effd640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-25T18:09:38.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.379+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbf84006ba0 con 0x7fbf900b14e0 2024-03-25T18:09:38.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.379+0000 7fbf7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:38.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.379+0000 7fbf7effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:38.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.451+0000 7fbf94ff6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b99f705f-8c5b-4b56-9e68-e6308b38ac09", "id": 5}] 2024-03-25T18:09:38.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.451+0000 7fbf94ff6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b99f705f-8c5b-4b56-9e68-e6308b38ac09", "id": 5}] 2024-03-25T18:09:38.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.451+0000 7fbf94ff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:38.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.451+0000 7fbf94ff6640 1 -- 172.21.15.86:0/2173581792 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b99f705f-8c5b-4b56-9e68-e6308b38ac09", "id": 5} v 0) v1 -- 0x7fbf9004f730 con 0x7fbf900b14e0 2024-03-25T18:09:38.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.530+0000 7fbf7effd640 1 -- 172.21.15.86:0/2173581792 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b99f705f-8c5b-4b56-9e68-e6308b38ac09", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2252304859 0 1988412176) 0x7fbf840063d0 con 0x7fbf900b14e0 2024-03-25T18:09:38.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.530+0000 7fbf7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b99f705f-8c5b-4b56-9e68-e6308b38ac09", "id": 5}] 2024-03-25T18:09:38.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.530+0000 7fbf7effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:38.531 INFO:teuthology.orchestra.run.smithi086.stdout:5 2024-03-25T18:09:38.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.531+0000 7fbf94ff6640 10 librados: watch_flush enter 2024-03-25T18:09:38.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.531+0000 7fbf94ff6640 10 librados: watch_flush exit 2024-03-25T18:09:38.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.531+0000 7fbf94ff6640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-25T18:09:38.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.531+0000 7fbf94ff6640 10 client.4150.objecter successfully canceled tick 2024-03-25T18:09:38.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.531+0000 7fbf94ff6640 10 monclient: shutdown 2024-03-25T18:09:38.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.531+0000 7fbf94ff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:38.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.531+0000 7fbf94ff6640 1 -- 172.21.15.86:0/2173581792 >> v1:172.21.15.86:6789/0 conn(0x7fbf900b14e0 legacy=0x7fbf900af150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:38.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.531+0000 7fbf94ff6640 1 -- 172.21.15.86:0/2173581792 shutdown_connections 2024-03-25T18:09:38.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.531+0000 7fbf94ff6640 1 -- 172.21.15.86:0/2173581792 >> 172.21.15.86:0/2173581792 conn(0x7fbf900b7df0 msgr2=0x7fbf900b81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:38.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.531+0000 7fbf94ff6640 1 -- 172.21.15.86:0/2173581792 shutdown_connections 2024-03-25T18:09:38.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.531+0000 7fbf94ff6640 1 -- 172.21.15.86:0/2173581792 wait complete. 2024-03-25T18:09:38.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.531+0000 7fbf94ff6640 1 librados: shutdown 2024-03-25T18:09:38.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a 6 2024-03-25T18:09:38.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.641+0000 7f8858315640 10 monclient: get_monmap_and_config 2024-03-25T18:09:38.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.641+0000 7f8858315640 10 monclient: build_initial_monmap 2024-03-25T18:09:38.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.642+0000 7f8858315640 10 monclient: monmap: 2024-03-25T18:09:38.643 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-25T18:09:38.643 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:38.643 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:09:38.643018+0000 2024-03-25T18:09:38.643 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:09:38.643018+0000 2024-03-25T18:09:38.643 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:38.643 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:38.643 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:38.644 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:09:38.644 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:09:38.644 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:38.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.642+0000 7f8858315640 1 Processor -- start 2024-03-25T18:09:38.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.642+0000 7f8858315640 1 -- start start 2024-03-25T18:09:38.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.642+0000 7f8858315640 10 monclient: init 2024-03-25T18:09:38.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.643+0000 7f8858315640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:38.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.643+0000 7f8858315640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:09:38.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.643+0000 7f8858315640 10 monclient(hunting): picked mon.noname-b con 0x7f88500b4700 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:38.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.643+0000 7f8858315640 10 monclient(hunting): picked mon.noname-a con 0x7f8850142a20 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:38.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.643+0000 7f8858315640 10 monclient(hunting): picked mon.noname-c con 0x7f88500a96b0 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:38.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.643+0000 7f8858315640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88500a9d60 con 0x7f8850142a20 2024-03-25T18:09:38.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.643+0000 7f8858315640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f885013ff10 con 0x7f88500a96b0 2024-03-25T18:09:38.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.643+0000 7f8858315640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88500aa1f0 con 0x7f88500b4700 2024-03-25T18:09:38.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.643+0000 7f8858315640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:38.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.643+0000 7f8858315640 10 monclient(hunting): authenticate will time out at 6959.880261s 2024-03-25T18:09:38.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.643+0000 7f88568b2640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f88500a96b0 0x7f885013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46472/0 (socket says 172.21.15.86:46472) 2024-03-25T18:09:38.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.643+0000 7f88568b2640 1 -- 172.21.15.86:0/942944651 learned_addr learned my addr 172.21.15.86:0/942944651 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-25T18:09:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.646+0000 7f88550af640 1 -- 172.21.15.86:0/942944651 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88391233 0 0) 0x7f88500a9d60 con 0x7f8850142a20 2024-03-25T18:09:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.646+0000 7f88550af640 10 monclient(hunting): _negotiate 2024-03-25T18:09:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.646+0000 7f88550af640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.646+0000 7f88550af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.646+0000 7f88550af640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:09:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.646+0000 7f88550af640 1 -- 172.21.15.86:0/942944651 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f883c003520 con 0x7f8850142a20 2024-03-25T18:09:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.646+0000 7f88550af640 1 -- 172.21.15.86:0/942944651 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708121899 0 0) 0x7f885013ff10 con 0x7f88500a96b0 2024-03-25T18:09:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.646+0000 7f88550af640 10 monclient(hunting): _negotiate 2024-03-25T18:09:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.646+0000 7f88550af640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.646+0000 7f88550af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.646+0000 7f88550af640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:09:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.646+0000 7f88550af640 1 -- 172.21.15.86:0/942944651 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88500a9d60 con 0x7f88500a96b0 2024-03-25T18:09:38.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.646+0000 7f88550af640 1 -- 172.21.15.86:0/942944651 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679883057 0 0) 0x7f88500aa1f0 con 0x7f88500b4700 2024-03-25T18:09:38.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.646+0000 7f88550af640 10 monclient(hunting): _negotiate 2024-03-25T18:09:38.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.646+0000 7f88550af640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:38.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.646+0000 7f88550af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:38.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.646+0000 7f88550af640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:09:38.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.646+0000 7f88550af640 1 -- 172.21.15.86:0/942944651 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f885013ff10 con 0x7f88500b4700 2024-03-25T18:09:38.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.649+0000 7f88550af640 1 -- 172.21.15.86:0/942944651 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3792336675 0 0) 0x7f88500a9d60 con 0x7f88500a96b0 2024-03-25T18:09:38.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.649+0000 7f88550af640 1 -- 172.21.15.86:0/942944651 >> v1:172.21.15.64:6790/0 conn(0x7f88500b4700 legacy=0x7f8850142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:38.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.649+0000 7f88550af640 1 -- 172.21.15.86:0/942944651 >> v1:172.21.15.64:6789/0 conn(0x7f8850142a20 legacy=0x7f885013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:38.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.649+0000 7f88550af640 10 monclient: _finish_hunting 0 2024-03-25T18:09:38.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.649+0000 7f88550af640 1 monclient: found mon.noname-c 2024-03-25T18:09:38.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.649+0000 7f88550af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:38.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.649+0000 7f88550af640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:09:38.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.649+0000 7f88550af640 1 -- 172.21.15.86:0/942944651 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88500b4ae0 con 0x7f88500a96b0 2024-03-25T18:09:38.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.649+0000 7f88550af640 10 monclient: _finish_auth 0 2024-03-25T18:09:38.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.649+0000 7f88550af640 10 monclient: _check_auth_tickets 2024-03-25T18:09:38.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.649+0000 7f88550af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:38.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.649+0000 7f88550af640 1 -- 172.21.15.86:0/942944651 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f88480019f0 con 0x7f88500a96b0 2024-03-25T18:09:38.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.649+0000 7f8858315640 5 monclient: authenticate success, global_id 4153 2024-03-25T18:09:38.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.649+0000 7f8858315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:38.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.649+0000 7f88550af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:38.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.649+0000 7f88550af640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:38.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.649+0000 7f88550af640 10 monclient: dump: 2024-03-25T18:09:38.652 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:38.652 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:38.652 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.652 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.652 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:38.652 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:38.652 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:38.652 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:38.652 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:38.652 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:38.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.649+0000 7f8858315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:38.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.651+0000 7f88550af640 1 -- 172.21.15.86:0/942944651 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8848001b50 con 0x7f88500a96b0 2024-03-25T18:09:38.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.651+0000 7f88550af640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:38.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.651+0000 7f8858315640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:38.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.651+0000 7f8858315640 4 set_mon_vals no callback set 2024-03-25T18:09:38.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.651+0000 7f88550af640 1 -- 172.21.15.86:0/942944651 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f8848001f60 con 0x7f88500a96b0 2024-03-25T18:09:38.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.651+0000 7f88550af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:38.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.651+0000 7f88550af640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:38.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.651+0000 7f88550af640 10 monclient: dump: 2024-03-25T18:09:38.656 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:38.656 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:38.656 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.656 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.656 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:38.656 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:38.656 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:38.656 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:38.656 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:38.656 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:38.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.651+0000 7f8858315640 10 monclient: shutdown 2024-03-25T18:09:38.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.651+0000 7f8858315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:38.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.651+0000 7f8858315640 1 -- 172.21.15.86:0/942944651 >> v1:172.21.15.86:6789/0 conn(0x7f88500a96b0 legacy=0x7f885013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:38.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.651+0000 7f8858315640 1 -- 172.21.15.86:0/942944651 shutdown_connections 2024-03-25T18:09:38.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.651+0000 7f8858315640 1 -- 172.21.15.86:0/942944651 >> 172.21.15.86:0/942944651 conn(0x7f88500668e0 msgr2=0x7f8850066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:38.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.651+0000 7f8858315640 1 -- 172.21.15.86:0/942944651 shutdown_connections 2024-03-25T18:09:38.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.651+0000 7f8858315640 1 -- 172.21.15.86:0/942944651 wait complete. 2024-03-25T18:09:38.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.652+0000 7f8858315640 10 monclient: build_initial_monmap 2024-03-25T18:09:38.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.652+0000 7f8858315640 10 monclient: monmap: 2024-03-25T18:09:38.657 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-25T18:09:38.657 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:38.657 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-03-25T18:09:38.657 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-03-25T18:09:38.657 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:38.657 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-25T18:09:38.657 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:38.657 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:09:38.657 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:09:38.658 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:38.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.652+0000 7f8858315640 1 librados: starting msgr at 2024-03-25T18:09:38.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.652+0000 7f8858315640 1 librados: starting objecter 2024-03-25T18:09:38.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.652+0000 7f8858315640 1 Processor -- start 2024-03-25T18:09:38.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.653+0000 7f8858315640 1 -- start start 2024-03-25T18:09:38.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.653+0000 7f8858315640 1 librados: setting wanted keys 2024-03-25T18:09:38.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.653+0000 7f8858315640 1 librados: calling monclient init 2024-03-25T18:09:38.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.653+0000 7f8858315640 10 monclient: init 2024-03-25T18:09:38.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.653+0000 7f8858315640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:38.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.653+0000 7f8858315640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:38.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.653+0000 7f8858315640 10 monclient(hunting): picked mon.noname-b con 0x7f88500a96b0 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:38.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.653+0000 7f8858315640 10 monclient(hunting): picked mon.noname-c con 0x7f88500b4700 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:38.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.653+0000 7f8858315640 10 monclient(hunting): picked mon.noname-a con 0x7f8850142a20 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:38.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.653+0000 7f8858315640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88500adce0 con 0x7f8850142a20 2024-03-25T18:09:38.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.653+0000 7f8858315640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88500b1c70 con 0x7f88500a96b0 2024-03-25T18:09:38.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.653+0000 7f8858315640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88501d4a40 con 0x7f88500b4700 2024-03-25T18:09:38.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.653+0000 7f8858315640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:38.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.653+0000 7f8858315640 10 monclient(hunting): authenticate will time out at 6959.890503s 2024-03-25T18:09:38.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.654+0000 7f88560b1640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f88500a96b0 0x7f88501d4240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46486/0 (socket says 172.21.15.86:46486) 2024-03-25T18:09:38.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.654+0000 7f88560b1640 1 -- 172.21.15.86:0/176127731 learned_addr learned my addr 172.21.15.86:0/176127731 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-25T18:09:38.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.654+0000 7f8842ffd640 10 client.?.objecter ms_handle_connect 0x7f88500a96b0 2024-03-25T18:09:38.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.654+0000 7f8842ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:38.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.654+0000 7f8842ffd640 10 client.?.objecter ms_handle_connect 0x7f8850142a20 2024-03-25T18:09:38.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.654+0000 7f8842ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:38.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.654+0000 7f8842ffd640 10 client.?.objecter ms_handle_connect 0x7f88500b4700 2024-03-25T18:09:38.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.654+0000 7f8842ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:38.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.656+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606840073 0 0) 0x7f88500adce0 con 0x7f8850142a20 2024-03-25T18:09:38.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.656+0000 7f8842ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:38.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.656+0000 7f8842ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:38.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.656+0000 7f8842ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:38.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.656+0000 7f8842ffd640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:09:38.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.656+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8834004eb0 con 0x7f8850142a20 2024-03-25T18:09:38.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.656+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2743783676 0 0) 0x7f88501d4a40 con 0x7f88500b4700 2024-03-25T18:09:38.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.656+0000 7f8842ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:38.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.656+0000 7f8842ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:38.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.656+0000 7f8842ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:38.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.656+0000 7f8842ffd640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:09:38.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.656+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88500adce0 con 0x7f88500b4700 2024-03-25T18:09:38.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.656+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312974109 0 0) 0x7f88500b1c70 con 0x7f88500a96b0 2024-03-25T18:09:38.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.656+0000 7f8842ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:38.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.656+0000 7f8842ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:38.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.656+0000 7f8842ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:38.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.656+0000 7f8842ffd640 10 monclient(hunting): my global_id is 4156 2024-03-25T18:09:38.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.656+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88501d4a40 con 0x7f88500a96b0 2024-03-25T18:09:38.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.660+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2130617546 0 0) 0x7f88500adce0 con 0x7f88500b4700 2024-03-25T18:09:38.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.660+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88500b1c70 con 0x7f88500b4700 2024-03-25T18:09:38.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.660+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f884c001770 con 0x7f88500b4700 2024-03-25T18:09:38.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.660+0000 7f8842ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:38.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.660+0000 7f8842ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:38.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.660+0000 7f8842ffd640 10 monclient(hunting): dump: 2024-03-25T18:09:38.663 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:38.663 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:38.663 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.663 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.664 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:38.664 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:38.664 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:38.664 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:38.664 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:38.664 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:38.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.660+0000 7f8842ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:38.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.661+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3566519188 0 0) 0x7f88501d4a40 con 0x7f88500a96b0 2024-03-25T18:09:38.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.661+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88500adce0 con 0x7f88500a96b0 2024-03-25T18:09:38.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.661+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f88440019b0 con 0x7f88500a96b0 2024-03-25T18:09:38.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.661+0000 7f8842ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:38.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.661+0000 7f8842ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:38.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.661+0000 7f8842ffd640 10 monclient(hunting): dump: 2024-03-25T18:09:38.664 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:38.664 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:38.665 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.665 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.665 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:38.665 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:38.665 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:38.665 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:38.665 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:38.665 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:38.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.661+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 934635134 0 0) 0x7f8834004eb0 con 0x7f8850142a20 2024-03-25T18:09:38.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.661+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88501d4a40 con 0x7f8850142a20 2024-03-25T18:09:38.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.661+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f8848001e60 con 0x7f8850142a20 2024-03-25T18:09:38.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.661+0000 7f8842ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:38.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.661+0000 7f8842ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:38.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.661+0000 7f8842ffd640 10 monclient(hunting): dump: 2024-03-25T18:09:38.665 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:38.665 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:38.666 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.666 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.666 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:38.666 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:38.666 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:38.666 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:38.666 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:38.666 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:38.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 459886569 0 0) 0x7f88501d4a40 con 0x7f8850142a20 2024-03-25T18:09:38.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 >> v1:172.21.15.64:6790/0 conn(0x7f88500b4700 legacy=0x7f88500b0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:38.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 >> v1:172.21.15.86:6789/0 conn(0x7f88500a96b0 legacy=0x7f88501d4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:38.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8842ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:09:38.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8842ffd640 1 monclient: found mon.a 2024-03-25T18:09:38.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8842ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:38.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8842ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:09:38.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88500b11d0 con 0x7f8850142a20 2024-03-25T18:09:38.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8842ffd640 10 monclient: _finish_auth 0 2024-03-25T18:09:38.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8842ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:09:38.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8842ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:38.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8858315640 5 monclient: authenticate success, global_id 4155 2024-03-25T18:09:38.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8858315640 10 monclient: _renew_subs 2024-03-25T18:09:38.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8858315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:09:38.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8858315640 1 -- 172.21.15.86:0/176127731 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88500b1480 con 0x7f8850142a20 2024-03-25T18:09:38.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8858315640 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:38.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8858315640 10 monclient: _renew_subs 2024-03-25T18:09:38.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8858315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:09:38.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8858315640 1 -- 172.21.15.86:0/176127731 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88500b0d40 con 0x7f8850142a20 2024-03-25T18:09:38.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.664+0000 7f8858315640 1 librados: init done 2024-03-25T18:09:38.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.665+0000 7f8858315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:38.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.665+0000 7f8858315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:38.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.665+0000 7f8858315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:09:38.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.665+0000 7f8858315640 1 -- 172.21.15.86:0/176127731 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88501d4c80 con 0x7f8850142a20 2024-03-25T18:09:38.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.669+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8848004e10 con 0x7f8850142a20 2024-03-25T18:09:38.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.669+0000 7f8842ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:38.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.669+0000 7f88548ae640 4 set_mon_vals no callback set 2024-03-25T18:09:38.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.850+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f8848004f80 con 0x7f8850142a20 2024-03-25T18:09:38.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.850+0000 7f8842ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:38.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.850+0000 7f8842ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:38.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.850+0000 7f8842ffd640 10 monclient: dump: 2024-03-25T18:09:38.851 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:38.851 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:38.852 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.852 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:38.852 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:38.852 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:38.852 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:38.852 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:38.852 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:38.852 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:38.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.850+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8848005470 con 0x7f8850142a20 2024-03-25T18:09:38.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.850+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 4103663504 0 0) 0x7f8848006b00 con 0x7f8850142a20 2024-03-25T18:09:38.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.850+0000 7f8842ffd640 10 client.4155.objecter ms_dispatch 0x7f8850000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:09:38.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.850+0000 7f8842ffd640 3 client.4155.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:09:38.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.850+0000 7f8842ffd640 3 client.4155.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:09:38.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.850+0000 7f8842ffd640 20 client.4155.objecter dump_active .. 0 homeless 2024-03-25T18:09:38.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.853+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f88480065c0 con 0x7f8850142a20 2024-03-25T18:09:38.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.853+0000 7f8842ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:38.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.853+0000 7f8842ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:38.874 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:38.874+0000 7fb779810040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:09:38.920 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:38.919+0000 7f7080038040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:09:38.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.926+0000 7f8858315640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a", "id": 6}] 2024-03-25T18:09:38.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.926+0000 7f8858315640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a", "id": 6}] 2024-03-25T18:09:38.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.926+0000 7f8858315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:09:38.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:38.926+0000 7f8858315640 1 -- 172.21.15.86:0/176127731 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a", "id": 6} v 0) v1 -- 0x7f885004f730 con 0x7f8850142a20 2024-03-25T18:09:39.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.048+0000 7f8842ffd640 1 -- 172.21.15.86:0/176127731 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2098302323 0 1113811849) 0x7f8848006850 con 0x7f8850142a20 2024-03-25T18:09:39.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.048+0000 7f8842ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a", "id": 6}] 2024-03-25T18:09:39.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.048+0000 7f8842ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:39.049 INFO:teuthology.orchestra.run.smithi086.stdout:6 2024-03-25T18:09:39.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.049+0000 7f8858315640 10 librados: watch_flush enter 2024-03-25T18:09:39.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.049+0000 7f8858315640 10 librados: watch_flush exit 2024-03-25T18:09:39.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.049+0000 7f8858315640 20 client.4155.objecter shutdown clearing up homeless session... 2024-03-25T18:09:39.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.049+0000 7f8858315640 10 client.4155.objecter successfully canceled tick 2024-03-25T18:09:39.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.049+0000 7f8858315640 10 monclient: shutdown 2024-03-25T18:09:39.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.049+0000 7f8858315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:39.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.049+0000 7f8858315640 1 -- 172.21.15.86:0/176127731 >> v1:172.21.15.64:6789/0 conn(0x7f8850142a20 legacy=0x7f88501d7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.049+0000 7f8858315640 1 -- 172.21.15.86:0/176127731 shutdown_connections 2024-03-25T18:09:39.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.049+0000 7f8858315640 1 -- 172.21.15.86:0/176127731 >> 172.21.15.86:0/176127731 conn(0x7f88500668e0 msgr2=0x7f885013d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:39.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.049+0000 7f8858315640 1 -- 172.21.15.86:0/176127731 shutdown_connections 2024-03-25T18:09:39.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.049+0000 7f8858315640 1 -- 172.21.15.86:0/176127731 wait complete. 2024-03-25T18:09:39.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.049+0000 7f8858315640 1 librados: shutdown 2024-03-25T18:09:39.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 4e25c52d-8e4e-4bb5-b982-d962bae4c3a7 7 2024-03-25T18:09:39.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.159+0000 7f91cd201640 10 monclient: get_monmap_and_config 2024-03-25T18:09:39.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.159+0000 7f91cd201640 10 monclient: build_initial_monmap 2024-03-25T18:09:39.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.159+0000 7f91cd201640 10 monclient: monmap: 2024-03-25T18:09:39.160 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-25T18:09:39.160 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:39.160 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:09:39.160058+0000 2024-03-25T18:09:39.160 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:09:39.160058+0000 2024-03-25T18:09:39.160 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:39.160 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:39.160 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:39.160 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:09:39.160 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:09:39.160 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:39.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.159+0000 7f91cd201640 1 Processor -- start 2024-03-25T18:09:39.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.159+0000 7f91cd201640 1 -- start start 2024-03-25T18:09:39.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.159+0000 7f91cd201640 10 monclient: init 2024-03-25T18:09:39.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.160+0000 7f91cd201640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:39.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.160+0000 7f91cd201640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:09:39.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.160+0000 7f91cd201640 10 monclient(hunting): picked mon.noname-b con 0x7f91c80b4a40 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:39.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.160+0000 7f91cd201640 10 monclient(hunting): picked mon.noname-a con 0x7f91c80afb10 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:39.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.160+0000 7f91cd201640 10 monclient(hunting): picked mon.noname-c con 0x7f91c813fce0 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:39.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.160+0000 7f91cd201640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c80b4490 con 0x7f91c80afb10 2024-03-25T18:09:39.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.160+0000 7f91cd201640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c80b3070 con 0x7f91c813fce0 2024-03-25T18:09:39.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.160+0000 7f91cd201640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c80b3200 con 0x7f91c80b4a40 2024-03-25T18:09:39.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.160+0000 7f91cd201640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:39.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.160+0000 7f91cd201640 10 monclient(hunting): authenticate will time out at 6960.397132s 2024-03-25T18:09:39.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.160+0000 7f91c659c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f91c80afb10 0x7f91c813f5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.86:42750/0 (socket says 172.21.15.86:42750) 2024-03-25T18:09:39.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.160+0000 7f91c659c640 1 -- 172.21.15.86:0/393951467 learned_addr learned my addr 172.21.15.86:0/393951467 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-25T18:09:39.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.163+0000 7f91c5d9b640 1 -- 172.21.15.86:0/393951467 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941726658 0 0) 0x7f91c80b4490 con 0x7f91c80afb10 2024-03-25T18:09:39.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.163+0000 7f91c5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.163+0000 7f91c5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.163+0000 7f91c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.163+0000 7f91c5d9b640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:09:39.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.163+0000 7f91c5d9b640 1 -- 172.21.15.86:0/393951467 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91b4003520 con 0x7f91c80afb10 2024-03-25T18:09:39.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.163+0000 7f91c5d9b640 1 -- 172.21.15.86:0/393951467 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657048972 0 0) 0x7f91c80b3070 con 0x7f91c813fce0 2024-03-25T18:09:39.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.163+0000 7f91c5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.163+0000 7f91c5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.163+0000 7f91c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.163+0000 7f91c5d9b640 10 monclient(hunting): my global_id is 4159 2024-03-25T18:09:39.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.163+0000 7f91c5d9b640 1 -- 172.21.15.86:0/393951467 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c80b4490 con 0x7f91c813fce0 2024-03-25T18:09:39.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.163+0000 7f91c5d9b640 1 -- 172.21.15.86:0/393951467 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129669476 0 0) 0x7f91c80b3200 con 0x7f91c80b4a40 2024-03-25T18:09:39.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.163+0000 7f91c5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.163+0000 7f91c5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.163+0000 7f91c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.163+0000 7f91c5d9b640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:09:39.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.163+0000 7f91c5d9b640 1 -- 172.21.15.86:0/393951467 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c80b3070 con 0x7f91c80b4a40 2024-03-25T18:09:39.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.166+0000 7f91c5d9b640 1 -- 172.21.15.86:0/393951467 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 735834931 0 0) 0x7f91b4003520 con 0x7f91c80afb10 2024-03-25T18:09:39.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.166+0000 7f91c5d9b640 1 -- 172.21.15.86:0/393951467 >> v1:172.21.15.64:6790/0 conn(0x7f91c80b4a40 legacy=0x7f91c80b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.166+0000 7f91c5d9b640 1 -- 172.21.15.86:0/393951467 >> v1:172.21.15.86:6789/0 conn(0x7f91c813fce0 legacy=0x7f91c80b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.166+0000 7f91c5d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:09:39.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.166+0000 7f91c5d9b640 1 monclient: found mon.noname-a 2024-03-25T18:09:39.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.166+0000 7f91c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:39.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.166+0000 7f91c5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:09:39.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.166+0000 7f91c5d9b640 1 -- 172.21.15.86:0/393951467 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c80b3390 con 0x7f91c80afb10 2024-03-25T18:09:39.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.166+0000 7f91c5d9b640 10 monclient: _finish_auth 0 2024-03-25T18:09:39.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.166+0000 7f91c5d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:09:39.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.166+0000 7f91c5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:39.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.166+0000 7f91c5d9b640 1 -- 172.21.15.86:0/393951467 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f91b8001690 con 0x7f91c80afb10 2024-03-25T18:09:39.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.166+0000 7f91c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.166+0000 7f91c5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:39.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.166+0000 7f91c5d9b640 10 monclient: dump: 2024-03-25T18:09:39.168 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:39.168 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:39.168 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:39.168 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:39.168 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.168 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:39.168 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:39.168 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:39.169 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:39.169 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:39.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.166+0000 7f91cd201640 5 monclient: authenticate success, global_id 4158 2024-03-25T18:09:39.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.166+0000 7f91cd201640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:39.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.168+0000 7f91c5d9b640 1 -- 172.21.15.86:0/393951467 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91b80017f0 con 0x7f91c80afb10 2024-03-25T18:09:39.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.168+0000 7f91c5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:39.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.168+0000 7f91c5d9b640 1 -- 172.21.15.86:0/393951467 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f91b8001c00 con 0x7f91c80afb10 2024-03-25T18:09:39.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.168+0000 7f91cd201640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:39.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.168+0000 7f91cd201640 4 set_mon_vals no callback set 2024-03-25T18:09:39.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.168+0000 7f91c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.168+0000 7f91c5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:39.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.168+0000 7f91c5d9b640 10 monclient: dump: 2024-03-25T18:09:39.170 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:39.170 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:39.170 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:39.170 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:39.171 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.171 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:39.171 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:39.171 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:39.171 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:39.171 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:39.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.169+0000 7f91cd201640 10 monclient: shutdown 2024-03-25T18:09:39.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.169+0000 7f91cd201640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:39.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.169+0000 7f91cd201640 1 -- 172.21.15.86:0/393951467 >> v1:172.21.15.64:6789/0 conn(0x7f91c80afb10 legacy=0x7f91c813f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.169+0000 7f91cd201640 1 -- 172.21.15.86:0/393951467 shutdown_connections 2024-03-25T18:09:39.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.169+0000 7f91cd201640 1 -- 172.21.15.86:0/393951467 >> 172.21.15.86:0/393951467 conn(0x7f91c80668e0 msgr2=0x7f91c8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:39.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.169+0000 7f91cd201640 1 -- 172.21.15.86:0/393951467 shutdown_connections 2024-03-25T18:09:39.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.169+0000 7f91cd201640 1 -- 172.21.15.86:0/393951467 wait complete. 2024-03-25T18:09:39.172 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:39.170+0000 7fb779810040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:09:39.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.170+0000 7f91cd201640 10 monclient: build_initial_monmap 2024-03-25T18:09:39.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.170+0000 7f91cd201640 10 monclient: monmap: 2024-03-25T18:09:39.174 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-25T18:09:39.174 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:39.174 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-03-25T18:09:39.174 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-03-25T18:09:39.174 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:39.174 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-25T18:09:39.174 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:39.174 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:09:39.174 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:09:39.174 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:39.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.170+0000 7f91cd201640 1 librados: starting msgr at 2024-03-25T18:09:39.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.170+0000 7f91cd201640 1 librados: starting objecter 2024-03-25T18:09:39.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.170+0000 7f91cd201640 1 Processor -- start 2024-03-25T18:09:39.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.170+0000 7f91cd201640 1 -- start start 2024-03-25T18:09:39.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.170+0000 7f91cd201640 1 librados: setting wanted keys 2024-03-25T18:09:39.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.170+0000 7f91cd201640 1 librados: calling monclient init 2024-03-25T18:09:39.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.170+0000 7f91cd201640 10 monclient: init 2024-03-25T18:09:39.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.171+0000 7f91cd201640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:39.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.171+0000 7f91cd201640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:09:39.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.171+0000 7f91cd201640 10 monclient(hunting): picked mon.noname-b con 0x7f91c80afb10 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:39.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.171+0000 7f91cd201640 10 monclient(hunting): picked mon.noname-a con 0x7f91c80b4a40 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:39.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.171+0000 7f91cd201640 10 monclient(hunting): picked mon.noname-c con 0x7f91c813fce0 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:39.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.171+0000 7f91cd201640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c81373a0 con 0x7f91c80b4a40 2024-03-25T18:09:39.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.171+0000 7f91cd201640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c8135ba0 con 0x7f91c80afb10 2024-03-25T18:09:39.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.171+0000 7f91cd201640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c8137610 con 0x7f91c813fce0 2024-03-25T18:09:39.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.171+0000 7f91cd201640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:39.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.171+0000 7f91cd201640 10 monclient(hunting): authenticate will time out at 6960.408079s 2024-03-25T18:09:39.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.171+0000 7f91c659c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f91c80b4a40 0x7f91c8137d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.86:42758/0 (socket says 172.21.15.86:42758) 2024-03-25T18:09:39.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.171+0000 7f91c659c640 1 -- 172.21.15.86:0/1327379114 learned_addr learned my addr 172.21.15.86:0/1327379114 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-25T18:09:39.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.171+0000 7f91bf7fe640 10 client.?.objecter ms_handle_connect 0x7f91c80b4a40 2024-03-25T18:09:39.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.171+0000 7f91bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:39.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.171+0000 7f91bf7fe640 10 client.?.objecter ms_handle_connect 0x7f91c80afb10 2024-03-25T18:09:39.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.171+0000 7f91bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:39.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.172+0000 7f91bf7fe640 10 client.?.objecter ms_handle_connect 0x7f91c813fce0 2024-03-25T18:09:39.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.172+0000 7f91bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:39.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.173+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267310486 0 0) 0x7f91c81373a0 con 0x7f91c80b4a40 2024-03-25T18:09:39.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.173+0000 7f91bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.173+0000 7f91bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.173+0000 7f91bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.173+0000 7f91bf7fe640 10 monclient(hunting): my global_id is 4161 2024-03-25T18:09:39.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.173+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a8004eb0 con 0x7f91c80b4a40 2024-03-25T18:09:39.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.174+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4207104204 0 0) 0x7f91c8135ba0 con 0x7f91c80afb10 2024-03-25T18:09:39.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.174+0000 7f91bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.174+0000 7f91bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.174+0000 7f91bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.174+0000 7f91bf7fe640 10 monclient(hunting): my global_id is 4162 2024-03-25T18:09:39.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.174+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c81373a0 con 0x7f91c80afb10 2024-03-25T18:09:39.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.174+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917530342 0 0) 0x7f91c8137610 con 0x7f91c813fce0 2024-03-25T18:09:39.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.174+0000 7f91bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.174+0000 7f91bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.174+0000 7f91bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.174+0000 7f91bf7fe640 10 monclient(hunting): my global_id is 4160 2024-03-25T18:09:39.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.174+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c8135ba0 con 0x7f91c813fce0 2024-03-25T18:09:39.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.178+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 804768300 0 0) 0x7f91c81373a0 con 0x7f91c80afb10 2024-03-25T18:09:39.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.178+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91c8137610 con 0x7f91c80afb10 2024-03-25T18:09:39.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.178+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f91b0001870 con 0x7f91c80afb10 2024-03-25T18:09:39.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.178+0000 7f91bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.178+0000 7f91bf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:39.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.178+0000 7f91bf7fe640 10 monclient(hunting): dump: 2024-03-25T18:09:39.179 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:39.179 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:39.179 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:39.179 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:39.179 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.179 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:39.179 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:39.179 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:39.179 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:39.180 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:39.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.178+0000 7f91bf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:39.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.179+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2668809384 0 0) 0x7f91a8004eb0 con 0x7f91c80b4a40 2024-03-25T18:09:39.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.179+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91c81373a0 con 0x7f91c80b4a40 2024-03-25T18:09:39.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.179+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f91b8001b40 con 0x7f91c80b4a40 2024-03-25T18:09:39.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.179+0000 7f91bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.179+0000 7f91bf7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:39.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.179+0000 7f91bf7fe640 10 monclient(hunting): dump: 2024-03-25T18:09:39.181 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:39.181 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:39.181 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:39.181 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:39.181 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.181 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:39.181 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:39.181 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:39.181 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:39.181 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:39.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.179+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1749933379 0 0) 0x7f91c8135ba0 con 0x7f91c813fce0 2024-03-25T18:09:39.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.179+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91a8004eb0 con 0x7f91c813fce0 2024-03-25T18:09:39.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.179+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f91c0001bd0 con 0x7f91c813fce0 2024-03-25T18:09:39.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.179+0000 7f91bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.179+0000 7f91bf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:39.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.179+0000 7f91bf7fe640 10 monclient(hunting): dump: 2024-03-25T18:09:39.182 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:39.182 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:39.182 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:39.182 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:39.182 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.182 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:39.182 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:39.182 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:39.182 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:39.182 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:39.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 188188726 0 0) 0x7f91c8137610 con 0x7f91c80afb10 2024-03-25T18:09:39.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 >> v1:172.21.15.64:6790/0 conn(0x7f91c813fce0 legacy=0x7f91c81e22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 >> v1:172.21.15.64:6789/0 conn(0x7f91c80b4a40 legacy=0x7f91c8137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91bf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:09:39.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91bf7fe640 1 monclient: found mon.b 2024-03-25T18:09:39.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:39.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91bf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:39.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c81378f0 con 0x7f91c80afb10 2024-03-25T18:09:39.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91bf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:09:39.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91bf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:09:39.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:39.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91cd201640 5 monclient: authenticate success, global_id 4162 2024-03-25T18:09:39.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91cd201640 10 monclient: _renew_subs 2024-03-25T18:09:39.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91cd201640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:39.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91cd201640 1 -- 172.21.15.86:0/1327379114 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c8138420 con 0x7f91c80afb10 2024-03-25T18:09:39.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91cd201640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:39.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91cd201640 10 monclient: _renew_subs 2024-03-25T18:09:39.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91cd201640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:39.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91cd201640 1 -- 172.21.15.86:0/1327379114 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c8135470 con 0x7f91c80afb10 2024-03-25T18:09:39.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.181+0000 7f91cd201640 1 librados: init done 2024-03-25T18:09:39.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.182+0000 7f91cd201640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:39.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.182+0000 7f91cd201640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:39.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.182+0000 7f91cd201640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:39.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.182+0000 7f91cd201640 1 -- 172.21.15.86:0/1327379114 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c80627b0 con 0x7f91c80afb10 2024-03-25T18:09:39.209 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:39.208+0000 7f7080038040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:09:39.227 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:39.226+0000 7fb779810040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:09:39.262 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:39.261+0000 7f7080038040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:09:39.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.281+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91b0003610 con 0x7f91c80afb10 2024-03-25T18:09:39.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.281+0000 7f91bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.281+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f91b0001d80 con 0x7f91c80afb10 2024-03-25T18:09:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.281+0000 7f91bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.281+0000 7f91bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.281+0000 7f91bf7fe640 10 monclient: dump: 2024-03-25T18:09:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-25T18:09:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 3 2024-03-25T18:09:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:39.283 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:09:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.281+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f91b0003cf0 con 0x7f91c80afb10 2024-03-25T18:09:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.281+0000 7f91c559a640 4 set_mon_vals no callback set 2024-03-25T18:09:39.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.281+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1478874081 0 0) 0x7f91b0005570 con 0x7f91c80afb10 2024-03-25T18:09:39.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.281+0000 7f91bf7fe640 10 client.4162.objecter ms_dispatch 0x7f91c8000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:09:39.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.281+0000 7f91bf7fe640 3 client.4162.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:09:39.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.281+0000 7f91bf7fe640 3 client.4162.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:09:39.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.281+0000 7f91bf7fe640 20 client.4162.objecter dump_active .. 0 homeless 2024-03-25T18:09:39.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.281+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f91b0005950 con 0x7f91c80afb10 2024-03-25T18:09:39.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.281+0000 7f91bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:39.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.281+0000 7f91bf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:39.347 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:39.347+0000 7fb779810040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:09:39.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.354+0000 7f91cd201640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4e25c52d-8e4e-4bb5-b982-d962bae4c3a7", "id": 7}] 2024-03-25T18:09:39.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.354+0000 7f91cd201640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4e25c52d-8e4e-4bb5-b982-d962bae4c3a7", "id": 7}] 2024-03-25T18:09:39.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.354+0000 7f91cd201640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:39.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.354+0000 7f91cd201640 1 -- 172.21.15.86:0/1327379114 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4e25c52d-8e4e-4bb5-b982-d962bae4c3a7", "id": 7} v 0) v1 -- 0x7f91c80ae9b0 con 0x7f91c80afb10 2024-03-25T18:09:39.382 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-25T18:09:39.381+0000 7f7080038040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:09:39.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.700+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 1914495637 0 0) 0x7f91b0004f10 con 0x7f91c80afb10 2024-03-25T18:09:39.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.848+0000 7f91bf7fe640 1 -- 172.21.15.86:0/1327379114 <== mon.1 v1:172.21.15.86:6789/0 11 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4e25c52d-8e4e-4bb5-b982-d962bae4c3a7", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2684312749 0 1371665406) 0x7f91b0014720 con 0x7f91c80afb10 2024-03-25T18:09:39.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.848+0000 7f91bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4e25c52d-8e4e-4bb5-b982-d962bae4c3a7", "id": 7}] 2024-03-25T18:09:39.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.848+0000 7f91bf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:39.850 INFO:teuthology.orchestra.run.smithi086.stdout:7 2024-03-25T18:09:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.849+0000 7f91cd201640 10 librados: watch_flush enter 2024-03-25T18:09:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.849+0000 7f91cd201640 10 librados: watch_flush exit 2024-03-25T18:09:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.849+0000 7f91cd201640 20 client.4162.objecter shutdown clearing up homeless session... 2024-03-25T18:09:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.849+0000 7f91cd201640 10 client.4162.objecter successfully canceled tick 2024-03-25T18:09:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.849+0000 7f91cd201640 10 monclient: shutdown 2024-03-25T18:09:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.849+0000 7f91cd201640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:39.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.849+0000 7f91cd201640 1 -- 172.21.15.86:0/1327379114 >> v1:172.21.15.86:6789/0 conn(0x7f91c80afb10 legacy=0x7f91c8136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.849+0000 7f91cd201640 1 -- 172.21.15.86:0/1327379114 shutdown_connections 2024-03-25T18:09:39.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.849+0000 7f91cd201640 1 -- 172.21.15.86:0/1327379114 >> 172.21.15.86:0/1327379114 conn(0x7f91c80668e0 msgr2=0x7f91c813f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:39.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.849+0000 7f91cd201640 1 -- 172.21.15.86:0/1327379114 shutdown_connections 2024-03-25T18:09:39.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.849+0000 7f91cd201640 1 -- 172.21.15.86:0/1327379114 wait complete. 2024-03-25T18:09:39.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-25T18:09:39.850+0000 7f91cd201640 1 librados: shutdown 2024-03-25T18:09:39.861 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:09:39.861 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:09:39.861 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:09:39.863 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:09:39.863 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:09:39.863 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:09:39.864 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:09:39.866 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:09:39.866 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:09:39.866 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:09:39.866 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:09:39.868 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:09:39.868 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:09:39.869 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:09:39.869 DEBUG:teuthology.orchestra.run.smithi064:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:09:39.871 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:09:39.872 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:09:39.872 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:09:39.872 DEBUG:teuthology.orchestra.run.smithi086:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:09:39.874 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:09:39.874 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:09:39.874 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:09:39.874 DEBUG:teuthology.orchestra.run.smithi086:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:09:39.876 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:09:39.876 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:09:39.876 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:09:39.877 DEBUG:teuthology.orchestra.run.smithi086:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:09:39.878 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:09:39.879 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:09:39.879 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:09:39.879 DEBUG:teuthology.orchestra.run.smithi086:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:09:39.881 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:09:39.881 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:40.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.019+0000 7f459cfed640 10 monclient: get_monmap_and_config 2024-03-25T18:09:40.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.019+0000 7f459cfed640 10 monclient: build_initial_monmap 2024-03-25T18:09:40.021 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.019+0000 7f459cfed640 10 monclient: monmap: 2024-03-25T18:09:40.021 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-03-25T18:09:40.021 INFO:teuthology.orchestra.run.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:40.021 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-25T18:09:40.020776+0000 2024-03-25T18:09:40.021 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-25T18:09:40.020776+0000 2024-03-25T18:09:40.021 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:40.021 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:40.022 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:40.022 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:09:40.022 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:09:40.022 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-25T18:09:40.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.020+0000 7f459cfed640 1 Processor -- start 2024-03-25T18:09:40.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.020+0000 7f459cfed640 1 -- start start 2024-03-25T18:09:40.022 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.020+0000 7f459cfed640 10 monclient: init 2024-03-25T18:09:40.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.021+0000 7f459cfed640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:40.023 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.021+0000 7f459cfed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:09:40.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.021+0000 7f459cfed640 10 monclient(hunting): picked mon.noname-b con 0x7f45980a9920 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:40.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.021+0000 7f459cfed640 10 monclient(hunting): picked mon.noname-a con 0x7f45980b45a0 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:40.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.021+0000 7f459cfed640 10 monclient(hunting): picked mon.noname-c con 0x7f4598146450 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:40.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.021+0000 7f459cfed640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459806b100 con 0x7f45980b45a0 2024-03-25T18:09:40.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.021+0000 7f459cfed640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45980b2940 con 0x7f4598146450 2024-03-25T18:09:40.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.021+0000 7f459cfed640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459813fec0 con 0x7f45980a9920 2024-03-25T18:09:40.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.021+0000 7f459cfed640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:40.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.022+0000 7f459cfed640 10 monclient(hunting): authenticate will time out at 6966.260473s 2024-03-25T18:09:40.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.022+0000 7f4595d9b640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f45980b45a0 0x7f4598145d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:45796/0 (socket says 172.21.15.64:45796) 2024-03-25T18:09:40.024 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.022+0000 7f4595d9b640 1 -- 172.21.15.64:0/2346446450 learned_addr learned my addr 172.21.15.64:0/2346446450 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:09:40.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.179+0000 7f459559a640 1 -- 172.21.15.64:0/2346446450 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420671457 0 0) 0x7f459806b100 con 0x7f45980b45a0 2024-03-25T18:09:40.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.179+0000 7f459559a640 10 monclient(hunting): _negotiate 2024-03-25T18:09:40.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.179+0000 7f459559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:40.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.179+0000 7f459559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:40.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.179+0000 7f459559a640 10 monclient(hunting): my global_id is 4173 2024-03-25T18:09:40.180 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.179+0000 7f459559a640 1 -- 172.21.15.64:0/2346446450 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4580003520 con 0x7f45980b45a0 2024-03-25T18:09:40.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.186+0000 7f459559a640 1 -- 172.21.15.64:0/2346446450 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3623311813 0 0) 0x7f4580003520 con 0x7f45980b45a0 2024-03-25T18:09:40.187 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.186+0000 7f459559a640 1 -- 172.21.15.64:0/2346446450 >> v1:172.21.15.64:6790/0 conn(0x7f45980a9920 legacy=0x7f459813f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:40.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.186+0000 7f459559a640 1 -- 172.21.15.64:0/2346446450 >> v1:172.21.15.86:6789/0 conn(0x7f4598146450 legacy=0x7f459814dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:40.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.186+0000 7f459559a640 10 monclient: _finish_hunting 0 2024-03-25T18:09:40.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.186+0000 7f459559a640 1 monclient: found mon.noname-a 2024-03-25T18:09:40.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.186+0000 7f459559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:40.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.186+0000 7f459559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:09:40.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.186+0000 7f459559a640 1 -- 172.21.15.64:0/2346446450 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4598140050 con 0x7f45980b45a0 2024-03-25T18:09:40.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.186+0000 7f459559a640 10 monclient: _finish_auth 0 2024-03-25T18:09:40.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.186+0000 7f459559a640 10 monclient: _check_auth_tickets 2024-03-25T18:09:40.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.186+0000 7f459559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:40.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.186+0000 7f459559a640 1 -- 172.21.15.64:0/2346446450 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f458c001690 con 0x7f45980b45a0 2024-03-25T18:09:40.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.186+0000 7f459cfed640 5 monclient: authenticate success, global_id 4173 2024-03-25T18:09:40.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.186+0000 7f459cfed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:40.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.186+0000 7f459559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:40.188 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.186+0000 7f459559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:40.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.186+0000 7f459559a640 10 monclient: dump: 2024-03-25T18:09:40.189 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-25T18:09:40.189 INFO:teuthology.orchestra.run.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:40.189 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:40.189 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:40.189 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:40.189 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:40.189 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:40.189 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:40.189 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:40.189 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-25T18:09:40.189 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.186+0000 7f459cfed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:40.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.202+0000 7f459559a640 1 -- 172.21.15.64:0/2346446450 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f458c0018b0 con 0x7f45980b45a0 2024-03-25T18:09:40.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.202+0000 7f459559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:40.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.202+0000 7f459559a640 1 -- 172.21.15.64:0/2346446450 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f458c001aa0 con 0x7f45980b45a0 2024-03-25T18:09:40.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.202+0000 7f459559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:40.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.202+0000 7f459559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:40.202 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.202+0000 7f459559a640 10 monclient: dump: 2024-03-25T18:09:40.203 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-25T18:09:40.203 INFO:teuthology.orchestra.run.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:40.203 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:40.203 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:40.203 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:40.203 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:40.203 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:40.203 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:40.203 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:40.203 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-25T18:09:40.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.202+0000 7f459cfed640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:40.203 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.202+0000 7f459cfed640 4 set_mon_vals no callback set 2024-03-25T18:09:40.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.202+0000 7f459cfed640 10 monclient: shutdown 2024-03-25T18:09:40.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.202+0000 7f459cfed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:40.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.202+0000 7f459cfed640 1 -- 172.21.15.64:0/2346446450 >> v1:172.21.15.64:6789/0 conn(0x7f45980b45a0 legacy=0x7f4598145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:40.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.203+0000 7f459cfed640 1 -- 172.21.15.64:0/2346446450 shutdown_connections 2024-03-25T18:09:40.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.203+0000 7f459cfed640 1 -- 172.21.15.64:0/2346446450 >> 172.21.15.64:0/2346446450 conn(0x7f45980b7f40 msgr2=0x7f45980b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:40.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.203+0000 7f459cfed640 1 -- 172.21.15.64:0/2346446450 shutdown_connections 2024-03-25T18:09:40.204 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.203+0000 7f459cfed640 1 -- 172.21.15.64:0/2346446450 wait complete. 2024-03-25T18:09:40.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.204+0000 7f459cfed640 10 monclient: build_initial_monmap 2024-03-25T18:09:40.205 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.204+0000 7f459cfed640 10 monclient: monmap: 2024-03-25T18:09:40.205 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 0 2024-03-25T18:09:40.205 INFO:teuthology.orchestra.run.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:40.206 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 0.000000 2024-03-25T18:09:40.206 INFO:teuthology.orchestra.run.smithi064.stderr:created 0.000000 2024-03-25T18:09:40.206 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:40.206 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 1 2024-03-25T18:09:40.206 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:40.206 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:09:40.206 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:09:40.206 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-25T18:09:40.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.205+0000 7f459cfed640 1 librados: starting msgr at 2024-03-25T18:09:40.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.205+0000 7f459cfed640 1 librados: starting objecter 2024-03-25T18:09:40.206 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.205+0000 7f459cfed640 1 Processor -- start 2024-03-25T18:09:40.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.205+0000 7f459cfed640 1 -- start start 2024-03-25T18:09:40.208 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.205+0000 7f459cfed640 1 librados: setting wanted keys 2024-03-25T18:09:40.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.205+0000 7f459cfed640 1 librados: calling monclient init 2024-03-25T18:09:40.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.205+0000 7f459cfed640 10 monclient: init 2024-03-25T18:09:40.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.206+0000 7f459cfed640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:40.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.206+0000 7f459cfed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:09:40.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.207+0000 7f459cfed640 10 monclient(hunting): picked mon.noname-b con 0x7f45980a9920 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:40.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.207+0000 7f459cfed640 10 monclient(hunting): picked mon.noname-a con 0x7f45980b45a0 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:40.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.207+0000 7f459cfed640 10 monclient(hunting): picked mon.noname-c con 0x7f4598146450 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:40.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.207+0000 7f459cfed640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4598142bf0 con 0x7f45980b45a0 2024-03-25T18:09:40.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.207+0000 7f459cfed640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459813bd00 con 0x7f45980a9920 2024-03-25T18:09:40.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.207+0000 7f459cfed640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459813be90 con 0x7f4598146450 2024-03-25T18:09:40.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.207+0000 7f4596d9d640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f4598146450 0x7f45981e1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:56048/0 (socket says 172.21.15.64:56048) 2024-03-25T18:09:40.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.207+0000 7f4596d9d640 1 -- 172.21.15.64:0/2860469679 learned_addr learned my addr 172.21.15.64:0/2860469679 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:09:40.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.207+0000 7f459cfed640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:40.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.207+0000 7f459cfed640 10 monclient(hunting): authenticate will time out at 6966.446449s 2024-03-25T18:09:40.209 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.208+0000 7f4586ffd640 10 client.?.objecter ms_handle_connect 0x7f4598146450 2024-03-25T18:09:40.210 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.208+0000 7f4586ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:40.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.210+0000 7f4586ffd640 1 -- 172.21.15.64:0/2860469679 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120756643 0 0) 0x7f459813be90 con 0x7f4598146450 2024-03-25T18:09:40.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.210+0000 7f4586ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:40.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.210+0000 7f4586ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:40.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.210+0000 7f4586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:40.211 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.210+0000 7f4586ffd640 10 monclient(hunting): my global_id is 4169 2024-03-25T18:09:40.212 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.210+0000 7f4586ffd640 1 -- 172.21.15.64:0/2860469679 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4578004eb0 con 0x7f4598146450 2024-03-25T18:09:40.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.212+0000 7f4586ffd640 10 client.?.objecter ms_handle_connect 0x7f45980b45a0 2024-03-25T18:09:40.213 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.212+0000 7f4586ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:40.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.218+0000 7f4586ffd640 10 client.?.objecter ms_handle_connect 0x7f45980a9920 2024-03-25T18:09:40.219 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.218+0000 7f4586ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:40.228 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.228+0000 7f4586ffd640 1 -- 172.21.15.64:0/2860469679 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2219558072 0 0) 0x7f4578004eb0 con 0x7f4598146450 2024-03-25T18:09:40.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.228+0000 7f4586ffd640 1 -- 172.21.15.64:0/2860469679 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f459813be90 con 0x7f4598146450 2024-03-25T18:09:40.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.228+0000 7f4586ffd640 1 -- 172.21.15.64:0/2860469679 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4590001c30 con 0x7f4598146450 2024-03-25T18:09:40.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.228+0000 7f4586ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:40.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.228+0000 7f4586ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:40.229 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.228+0000 7f4586ffd640 10 monclient(hunting): dump: 2024-03-25T18:09:40.229 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-25T18:09:40.229 INFO:teuthology.orchestra.run.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:40.229 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:40.229 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:40.229 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:40.229 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:40.229 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:40.229 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:40.229 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:40.229 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-25T18:09:40.230 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.228+0000 7f4586ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:40.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.309+0000 7f4586ffd640 1 -- 172.21.15.64:0/2860469679 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 596937339 0 0) 0x7f459813be90 con 0x7f4598146450 2024-03-25T18:09:40.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.309+0000 7f4586ffd640 1 -- 172.21.15.64:0/2860469679 >> v1:172.21.15.86:6789/0 conn(0x7f45980a9920 legacy=0x7f45981d70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:40.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.309+0000 7f4586ffd640 1 -- 172.21.15.64:0/2860469679 >> v1:172.21.15.64:6789/0 conn(0x7f45980b45a0 legacy=0x7f45981d77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:40.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.309+0000 7f4586ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:09:40.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.309+0000 7f4586ffd640 1 monclient: found mon.c 2024-03-25T18:09:40.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.309+0000 7f4586ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:40.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.309+0000 7f4586ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:40.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.309+0000 7f4586ffd640 1 -- 172.21.15.64:0/2860469679 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459813c020 con 0x7f4598146450 2024-03-25T18:09:40.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.309+0000 7f4586ffd640 10 monclient: _finish_auth 0 2024-03-25T18:09:40.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.309+0000 7f4586ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:09:40.311 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.309+0000 7f4586ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:40.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.310+0000 7f459cfed640 5 monclient: authenticate success, global_id 4169 2024-03-25T18:09:40.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.310+0000 7f459cfed640 10 monclient: _renew_subs 2024-03-25T18:09:40.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.310+0000 7f459cfed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:40.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.310+0000 7f459cfed640 1 -- 172.21.15.64:0/2860469679 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45981419e0 con 0x7f4598146450 2024-03-25T18:09:40.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.310+0000 7f459cfed640 10 client.4169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:40.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.310+0000 7f459cfed640 10 monclient: _renew_subs 2024-03-25T18:09:40.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.310+0000 7f459cfed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:40.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.310+0000 7f459cfed640 1 -- 172.21.15.64:0/2860469679 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4598142490 con 0x7f4598146450 2024-03-25T18:09:40.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.310+0000 7f459cfed640 1 librados: init done 2024-03-25T18:09:40.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.311+0000 7f459cfed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:40.312 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.311+0000 7f459cfed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:40.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.311+0000 7f459cfed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:40.313 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.311+0000 7f459cfed640 1 -- 172.21.15.64:0/2860469679 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f459804f730 con 0x7f4598146450 2024-03-25T18:09:40.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.315+0000 7f4586ffd640 1 -- 172.21.15.64:0/2860469679 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4590003970 con 0x7f4598146450 2024-03-25T18:09:40.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.315+0000 7f4586ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:40.315 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.315+0000 7f4594d99640 4 set_mon_vals no callback set 2024-03-25T18:09:40.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.496+0000 7f4586ffd640 1 -- 172.21.15.64:0/2860469679 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4590003dc0 con 0x7f4598146450 2024-03-25T18:09:40.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.496+0000 7f4586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:40.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.496+0000 7f4586ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:40.497 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.496+0000 7f4586ffd640 10 monclient: dump: 2024-03-25T18:09:40.497 INFO:teuthology.orchestra.run.smithi064.stderr:epoch 1 2024-03-25T18:09:40.497 INFO:teuthology.orchestra.run.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:40.497 INFO:teuthology.orchestra.run.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:40.497 INFO:teuthology.orchestra.run.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:40.497 INFO:teuthology.orchestra.run.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:40.497 INFO:teuthology.orchestra.run.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:40.497 INFO:teuthology.orchestra.run.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:40.497 INFO:teuthology.orchestra.run.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:40.498 INFO:teuthology.orchestra.run.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:40.498 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-25T18:09:40.503 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.503+0000 7f4586ffd640 1 -- 172.21.15.64:0/2860469679 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 1914495637 0 0) 0x7f4590003fa0 con 0x7f4598146450 2024-03-25T18:09:40.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.552+0000 7f4586ffd640 1 -- 172.21.15.64:0/2860469679 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f4590015a00 con 0x7f4598146450 2024-03-25T18:09:40.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.552+0000 7f4586ffd640 10 client.4169.objecter ms_dispatch 0x7f4598000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:40.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.552+0000 7f4586ffd640 3 client.4169.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:40.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.552+0000 7f4586ffd640 3 client.4169.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:40.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.552+0000 7f4586ffd640 20 client.4169.objecter dump_active .. 0 homeless 2024-03-25T18:09:40.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.552+0000 7f4586ffd640 1 -- 172.21.15.64:0/2860469679 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4590014cc0 con 0x7f4598146450 2024-03-25T18:09:40.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.552+0000 7f4586ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:40.553 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.552+0000 7f4586ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:40.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.676+0000 7f459cfed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:40.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.676+0000 7f459cfed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:40.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.676+0000 7f459cfed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:40.677 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.676+0000 7f459cfed640 1 -- 172.21.15.64:0/2860469679 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4598144bc0 con 0x7f4598146450 2024-03-25T18:09:40.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.842+0000 7f4586ffd640 1 -- 172.21.15.64:0/2860469679 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 1454723164 0 0) 0x7f4590004680 con 0x7f4598146450 2024-03-25T18:09:40.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.843+0000 7f459659c640 10 monclient: get_auth_request con 0x7f457806e220 auth_method 0 2024-03-25T18:09:40.843 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.843+0000 7f4586ffd640 10 client.4169.objecter ms_handle_connect 0x7f457806e220 2024-03-25T18:09:40.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.898+0000 7f4586ffd640 1 -- 172.21.15.64:0/2860469679 <== mon.2 v1:172.21.15.64:6790/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f4590009f90 con 0x7f4598146450 2024-03-25T18:09:40.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.899+0000 7f4586ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:40.899 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.899+0000 7f4586ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:40.899 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-25T18:09:40.900 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:40.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.900+0000 7f459cfed640 10 librados: watch_flush enter 2024-03-25T18:09:40.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.900+0000 7f459cfed640 10 librados: watch_flush exit 2024-03-25T18:09:40.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.900+0000 7f459cfed640 20 client.4169.objecter shutdown clearing up homeless session... 2024-03-25T18:09:40.901 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.900+0000 7f459cfed640 10 client.4169.objecter successfully canceled tick 2024-03-25T18:09:40.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.900+0000 7f459cfed640 1 -- 172.21.15.64:0/2860469679 >> v1:172.21.15.64:6800/2915543638 conn(0x7f457806e220 legacy=0x7f4578070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:40.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.900+0000 7f459cfed640 10 monclient: shutdown 2024-03-25T18:09:40.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.900+0000 7f459cfed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:40.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.900+0000 7f459cfed640 1 -- 172.21.15.64:0/2860469679 >> v1:172.21.15.64:6790/0 conn(0x7f4598146450 legacy=0x7f45981e1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:40.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.900+0000 7f459cfed640 1 -- 172.21.15.64:0/2860469679 shutdown_connections 2024-03-25T18:09:40.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.900+0000 7f459cfed640 1 -- 172.21.15.64:0/2860469679 >> 172.21.15.64:0/2860469679 conn(0x7f45980b7f40 msgr2=0x7f4598074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:40.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.901+0000 7f459cfed640 1 -- 172.21.15.64:0/2860469679 shutdown_connections 2024-03-25T18:09:40.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.901+0000 7f459cfed640 1 -- 172.21.15.64:0/2860469679 wait complete. 2024-03-25T18:09:40.902 INFO:teuthology.orchestra.run.smithi064.stderr:2024-03-25T18:09:40.901+0000 7f459cfed640 1 librados: shutdown 2024-03-25T18:09:40.916 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:09:40.916 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:09:40.946 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:40.945+0000 7fb71a0d1640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:09:41.189 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:09:41.189+0000 7fb71a0d1640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:09:41.722 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:41.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.870+0000 7ff55ef5d640 10 monclient: get_monmap_and_config 2024-03-25T18:09:41.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.871+0000 7ff55ef5d640 10 monclient: build_initial_monmap 2024-03-25T18:09:41.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.871+0000 7ff55ef5d640 10 monclient: monmap: 2024-03-25T18:09:41.872 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:09:41.872 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:41.872 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:09:41.871930+0000 2024-03-25T18:09:41.872 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:09:41.871930+0000 2024-03-25T18:09:41.872 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:41.872 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:41.872 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:41.872 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:09:41.872 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:09:41.872 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:41.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.872+0000 7ff55ef5d640 1 Processor -- start 2024-03-25T18:09:41.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.872+0000 7ff55ef5d640 1 -- start start 2024-03-25T18:09:41.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.872+0000 7ff55ef5d640 10 monclient: init 2024-03-25T18:09:41.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.873+0000 7ff55ef5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:41.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.873+0000 7ff55ef5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:09:41.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.873+0000 7ff55ef5d640 10 monclient(hunting): picked mon.noname-c con 0x7ff5580a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:41.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.873+0000 7ff55ef5d640 10 monclient(hunting): picked mon.noname-a con 0x7ff55813f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:41.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.873+0000 7ff55ef5d640 10 monclient(hunting): picked mon.noname-b con 0x7ff5581430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:41.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.873+0000 7ff55ef5d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff558142520 con 0x7ff55813f710 2024-03-25T18:09:41.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.873+0000 7ff55ef5d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5580b28a0 con 0x7ff5580a98f0 2024-03-25T18:09:41.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.873+0000 7ff55ef5d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5580b2a30 con 0x7ff5581430c0 2024-03-25T18:09:41.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.873+0000 7ff55ef5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:41.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.873+0000 7ff55ef5d640 10 monclient(hunting): authenticate will time out at 6968.112166s 2024-03-25T18:09:41.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.874+0000 7ff557fff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7ff55813f710 0x7ff5581429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:45830/0 (socket says 172.21.15.64:45830) 2024-03-25T18:09:41.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.874+0000 7ff557fff640 1 -- 172.21.15.64:0/1762582678 learned_addr learned my addr 172.21.15.64:0/1762582678 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:09:41.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.932+0000 7ff5577fe640 1 -- 172.21.15.64:0/1762582678 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098077468 0 0) 0x7ff5580b2a30 con 0x7ff5581430c0 2024-03-25T18:09:41.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.932+0000 7ff5577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:41.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.932+0000 7ff5577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:41.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.932+0000 7ff5577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:41.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.932+0000 7ff5577fe640 10 monclient(hunting): my global_id is 4175 2024-03-25T18:09:41.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.932+0000 7ff5577fe640 1 -- 172.21.15.64:0/1762582678 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff548003520 con 0x7ff5581430c0 2024-03-25T18:09:41.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.938+0000 7ff5577fe640 1 -- 172.21.15.64:0/1762582678 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3418574396 0 0) 0x7ff548003520 con 0x7ff5581430c0 2024-03-25T18:09:41.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.938+0000 7ff5577fe640 1 -- 172.21.15.64:0/1762582678 >> v1:172.21.15.86:6789/0 conn(0x7ff5580a98f0 legacy=0x7ff5580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:41.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.938+0000 7ff5577fe640 1 -- 172.21.15.64:0/1762582678 >> v1:172.21.15.64:6789/0 conn(0x7ff55813f710 legacy=0x7ff5581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:41.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.938+0000 7ff5577fe640 10 monclient: _finish_hunting 0 2024-03-25T18:09:41.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.938+0000 7ff5577fe640 1 monclient: found mon.noname-b 2024-03-25T18:09:41.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.938+0000 7ff5577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:41.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.938+0000 7ff5577fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:09:41.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.938+0000 7ff5577fe640 1 -- 172.21.15.64:0/1762582678 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5580b44c0 con 0x7ff5581430c0 2024-03-25T18:09:41.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.938+0000 7ff5577fe640 10 monclient: _finish_auth 0 2024-03-25T18:09:41.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.938+0000 7ff5577fe640 10 monclient: _check_auth_tickets 2024-03-25T18:09:41.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.938+0000 7ff5577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:41.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.938+0000 7ff5577fe640 1 -- 172.21.15.64:0/1762582678 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ff550001a10 con 0x7ff5581430c0 2024-03-25T18:09:41.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.938+0000 7ff5577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:41.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.938+0000 7ff5577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:41.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.938+0000 7ff5577fe640 10 monclient: dump: 2024-03-25T18:09:41.940 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:09:41.940 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:41.940 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:41.940 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:41.941 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:41.941 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:41.941 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:41.941 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:41.941 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:41.941 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:41.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.938+0000 7ff55ef5d640 5 monclient: authenticate success, global_id 4175 2024-03-25T18:09:41.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.938+0000 7ff55ef5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:41.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.941+0000 7ff5577fe640 1 -- 172.21.15.64:0/1762582678 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff550001b70 con 0x7ff5581430c0 2024-03-25T18:09:41.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.941+0000 7ff5577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:41.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.941+0000 7ff5577fe640 1 -- 172.21.15.64:0/1762582678 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ff550001fc0 con 0x7ff5581430c0 2024-03-25T18:09:41.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.941+0000 7ff5577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:41.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.941+0000 7ff5577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:41.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.941+0000 7ff5577fe640 10 monclient: dump: 2024-03-25T18:09:41.942 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:09:41.942 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:41.942 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:41.943 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:41.943 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:41.943 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:41.943 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:41.943 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:41.943 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:41.943 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:41.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.941+0000 7ff55ef5d640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:41.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.941+0000 7ff55ef5d640 4 set_mon_vals no callback set 2024-03-25T18:09:41.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.942+0000 7ff55ef5d640 10 monclient: shutdown 2024-03-25T18:09:41.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.942+0000 7ff55ef5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:41.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.942+0000 7ff55ef5d640 1 -- 172.21.15.64:0/1762582678 >> v1:172.21.15.64:6790/0 conn(0x7ff5581430c0 legacy=0x7ff55814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:41.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.942+0000 7ff55ef5d640 1 -- 172.21.15.64:0/1762582678 shutdown_connections 2024-03-25T18:09:41.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.942+0000 7ff55ef5d640 1 -- 172.21.15.64:0/1762582678 >> 172.21.15.64:0/1762582678 conn(0x7ff5580b7ea0 msgr2=0x7ff5580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:41.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.942+0000 7ff55ef5d640 1 -- 172.21.15.64:0/1762582678 shutdown_connections 2024-03-25T18:09:41.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.943+0000 7ff55ef5d640 1 -- 172.21.15.64:0/1762582678 wait complete. 2024-03-25T18:09:41.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.943+0000 7ff55ef5d640 10 monclient: build_initial_monmap 2024-03-25T18:09:41.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.943+0000 7ff55ef5d640 10 monclient: monmap: 2024-03-25T18:09:41.945 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:09:41.945 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:41.945 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:09:41.945 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:09:41.945 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:41.945 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:09:41.945 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:41.945 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:09:41.945 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:09:41.945 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:41.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.944+0000 7ff55ef5d640 1 librados: starting msgr at 2024-03-25T18:09:41.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.944+0000 7ff55ef5d640 1 librados: starting objecter 2024-03-25T18:09:41.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.944+0000 7ff55ef5d640 1 Processor -- start 2024-03-25T18:09:41.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.944+0000 7ff55ef5d640 1 -- start start 2024-03-25T18:09:41.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.944+0000 7ff55ef5d640 1 librados: setting wanted keys 2024-03-25T18:09:41.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.944+0000 7ff55ef5d640 1 librados: calling monclient init 2024-03-25T18:09:41.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.945+0000 7ff55ef5d640 10 monclient: init 2024-03-25T18:09:41.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.946+0000 7ff55ef5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:41.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.946+0000 7ff55ef5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:09:41.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.946+0000 7ff55ef5d640 10 monclient(hunting): picked mon.noname-a con 0x7ff5580a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:41.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.946+0000 7ff55ef5d640 10 monclient(hunting): picked mon.noname-b con 0x7ff55813f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:41.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.946+0000 7ff55ef5d640 10 monclient(hunting): picked mon.noname-c con 0x7ff5581430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:41.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.946+0000 7ff55ef5d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff558136ec0 con 0x7ff5580a98f0 2024-03-25T18:09:41.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.946+0000 7ff55ef5d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff558136c20 con 0x7ff55813f710 2024-03-25T18:09:41.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.946+0000 7ff55ef5d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff558137db0 con 0x7ff5581430c0 2024-03-25T18:09:41.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.946+0000 7ff55ef5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:41.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.946+0000 7ff55ef5d640 10 monclient(hunting): authenticate will time out at 6968.185252s 2024-03-25T18:09:41.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.947+0000 7ff55d4fa640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7ff5581430c0 0x7ff5581e2c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:56074/0 (socket says 172.21.15.64:56074) 2024-03-25T18:09:41.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.947+0000 7ff55d4fa640 1 -- 172.21.15.64:0/83233339 learned_addr learned my addr 172.21.15.64:0/83233339 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:09:41.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.947+0000 7ff5557fa640 10 client.?.objecter ms_handle_connect 0x7ff5581430c0 2024-03-25T18:09:41.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.947+0000 7ff5557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:41.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.950+0000 7ff5557fa640 1 -- 172.21.15.64:0/83233339 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221213382 0 0) 0x7ff558137db0 con 0x7ff5581430c0 2024-03-25T18:09:41.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.950+0000 7ff5557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:09:41.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.950+0000 7ff5557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:41.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.950+0000 7ff5557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:41.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.950+0000 7ff5557fa640 10 monclient(hunting): my global_id is 4178 2024-03-25T18:09:41.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.950+0000 7ff5557fa640 1 -- 172.21.15.64:0/83233339 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff53c004eb0 con 0x7ff5581430c0 2024-03-25T18:09:41.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.958+0000 7ff5557fa640 1 -- 172.21.15.64:0/83233339 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4009133455 0 0) 0x7ff53c004eb0 con 0x7ff5581430c0 2024-03-25T18:09:41.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.958+0000 7ff5557fa640 1 -- 172.21.15.64:0/83233339 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff558137db0 con 0x7ff5581430c0 2024-03-25T18:09:41.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.958+0000 7ff5557fa640 1 -- 172.21.15.64:0/83233339 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ff550001fc0 con 0x7ff5581430c0 2024-03-25T18:09:41.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.958+0000 7ff5557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:41.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.958+0000 7ff5557fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:41.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.958+0000 7ff5557fa640 10 monclient(hunting): dump: 2024-03-25T18:09:41.959 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:09:41.959 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:41.959 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:41.959 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:41.959 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:41.959 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:41.959 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:41.959 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:41.959 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:41.959 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:41.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.958+0000 7ff5557fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:41.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.965+0000 7ff5557fa640 1 -- 172.21.15.64:0/83233339 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1504203327 0 0) 0x7ff558137db0 con 0x7ff5581430c0 2024-03-25T18:09:41.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.965+0000 7ff5557fa640 1 -- 172.21.15.64:0/83233339 >> v1:172.21.15.86:6789/0 conn(0x7ff55813f710 legacy=0x7ff5580b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:41.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.965+0000 7ff5557fa640 1 -- 172.21.15.64:0/83233339 >> v1:172.21.15.64:6789/0 conn(0x7ff5580a98f0 legacy=0x7ff5581376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:41.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.965+0000 7ff5557fa640 10 monclient: _finish_hunting 0 2024-03-25T18:09:41.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.965+0000 7ff5557fa640 1 monclient: found mon.c 2024-03-25T18:09:41.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.965+0000 7ff5557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:41.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.965+0000 7ff5557fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:41.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.965+0000 7ff5557fa640 1 -- 172.21.15.64:0/83233339 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff558138b50 con 0x7ff5581430c0 2024-03-25T18:09:41.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.965+0000 7ff5557fa640 10 monclient: _finish_auth 0 2024-03-25T18:09:41.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.965+0000 7ff5557fa640 10 monclient: _check_auth_tickets 2024-03-25T18:09:41.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.965+0000 7ff5557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:41.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.966+0000 7ff55ef5d640 5 monclient: authenticate success, global_id 4178 2024-03-25T18:09:41.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.966+0000 7ff55ef5d640 10 monclient: _renew_subs 2024-03-25T18:09:41.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.966+0000 7ff55ef5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:41.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.966+0000 7ff55ef5d640 1 -- 172.21.15.64:0/83233339 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff558139830 con 0x7ff5581430c0 2024-03-25T18:09:41.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.966+0000 7ff55ef5d640 10 client.4178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:41.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.966+0000 7ff55ef5d640 10 monclient: _renew_subs 2024-03-25T18:09:41.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.966+0000 7ff55ef5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:41.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.966+0000 7ff55ef5d640 1 -- 172.21.15.64:0/83233339 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5581369b0 con 0x7ff5581430c0 2024-03-25T18:09:41.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.966+0000 7ff55ef5d640 1 librados: init done 2024-03-25T18:09:41.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.967+0000 7ff55ef5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:41.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.967+0000 7ff55ef5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:41.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.967+0000 7ff55ef5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:41.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:41.967+0000 7ff55ef5d640 1 -- 172.21.15.64:0/83233339 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff558139a20 con 0x7ff5581430c0 2024-03-25T18:09:42.018 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.018+0000 7ff5557fa640 1 -- 172.21.15.64:0/83233339 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff550003c20 con 0x7ff5581430c0 2024-03-25T18:09:42.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.018+0000 7ff5557fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:42.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.018+0000 7ff5557fa640 1 -- 172.21.15.64:0/83233339 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ff550003e90 con 0x7ff5581430c0 2024-03-25T18:09:42.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.018+0000 7ff5557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:42.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.018+0000 7ff5557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:42.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.018+0000 7ff5557fa640 10 monclient: dump: 2024-03-25T18:09:42.019 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:09:42.019 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:42.019 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:42.019 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:42.019 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:42.020 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:42.020 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:42.020 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:42.020 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:42.020 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:42.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.018+0000 7ff5557fa640 1 -- 172.21.15.64:0/83233339 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 1454723164 0 0) 0x7ff5500040b0 con 0x7ff5581430c0 2024-03-25T18:09:42.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.018+0000 7ff556ffd640 4 set_mon_vals no callback set 2024-03-25T18:09:42.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.020+0000 7ff55ccf9640 10 monclient: get_auth_request con 0x7ff53c06dad0 auth_method 0 2024-03-25T18:09:42.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.020+0000 7ff5557fa640 10 client.4178.objecter ms_handle_connect 0x7ff53c06dad0 2024-03-25T18:09:42.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.429+0000 7ff5557fa640 1 -- 172.21.15.64:0/83233339 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7ff550054c30 con 0x7ff5581430c0 2024-03-25T18:09:42.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.429+0000 7ff5557fa640 10 client.4178.objecter ms_dispatch 0x7ff558000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:42.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.429+0000 7ff5557fa640 3 client.4178.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:42.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.429+0000 7ff5557fa640 3 client.4178.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:42.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.429+0000 7ff5557fa640 20 client.4178.objecter dump_active .. 0 homeless 2024-03-25T18:09:42.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.429+0000 7ff5557fa640 1 -- 172.21.15.64:0/83233339 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff550050740 con 0x7ff5581430c0 2024-03-25T18:09:42.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.429+0000 7ff5557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:42.518 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.429+0000 7ff5557fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:42.577 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.576+0000 7ff5557fa640 1 -- 172.21.15.64:0/83233339 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7ff5500501f0 con 0x7ff5581430c0 2024-03-25T18:09:42.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.776+0000 7ff55ef5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:42.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.776+0000 7ff55ef5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:42.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.776+0000 7ff55ef5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:42.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.776+0000 7ff55ef5d640 1 -- 172.21.15.64:0/83233339 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5581373c0 con 0x7ff5581430c0 2024-03-25T18:09:42.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.803+0000 7ff5557fa640 1 -- 172.21.15.64:0/83233339 <== mon.2 v1:172.21.15.64:6790/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7ff5581373c0 con 0x7ff5581430c0 2024-03-25T18:09:42.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.803+0000 7ff5557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:42.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.803+0000 7ff5557fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:42.804 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:09:42.804 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:42.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.806+0000 7ff55ef5d640 10 librados: watch_flush enter 2024-03-25T18:09:42.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.806+0000 7ff55ef5d640 10 librados: watch_flush exit 2024-03-25T18:09:42.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.806+0000 7ff55ef5d640 20 client.4178.objecter shutdown clearing up homeless session... 2024-03-25T18:09:42.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.806+0000 7ff55ef5d640 10 client.4178.objecter successfully canceled tick 2024-03-25T18:09:42.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.806+0000 7ff55ef5d640 1 -- 172.21.15.64:0/83233339 >> v1:172.21.15.64:6800/2915543638 conn(0x7ff53c06dad0 legacy=0x7ff53c06ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:42.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.806+0000 7ff55ef5d640 10 monclient: shutdown 2024-03-25T18:09:42.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.806+0000 7ff55ef5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:42.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.806+0000 7ff55ef5d640 1 -- 172.21.15.64:0/83233339 >> v1:172.21.15.64:6790/0 conn(0x7ff5581430c0 legacy=0x7ff5581e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:42.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.806+0000 7ff55ef5d640 1 -- 172.21.15.64:0/83233339 shutdown_connections 2024-03-25T18:09:42.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.806+0000 7ff55ef5d640 1 -- 172.21.15.64:0/83233339 >> 172.21.15.64:0/83233339 conn(0x7ff5580b7ea0 msgr2=0x7ff55813ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:42.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.807+0000 7ff55ef5d640 1 -- 172.21.15.64:0/83233339 shutdown_connections 2024-03-25T18:09:42.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.807+0000 7ff55ef5d640 1 -- 172.21.15.64:0/83233339 wait complete. 2024-03-25T18:09:42.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:42.807+0000 7ff55ef5d640 1 librados: shutdown 2024-03-25T18:09:42.822 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:46.646 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:00:06.699 35764== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:46.713 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:00:06.766 35764== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:46.752 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:00:06.809 35763== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:46.796 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:00:06.847 35765== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:46.814 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:00:06.870 35763== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:46.855 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:00:06.907 35765== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:46.876 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:00:06.933 35762== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:46.936 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:00:06.992 35762== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:47.263 INFO:tasks.ceph.osd.7.smithi086.stderr:2024-03-25T18:09:47.256+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:09:47.335 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:07.381 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:47.339 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:07.393 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:47.370 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-03-25T18:09:47.364+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:09:47.401 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:07.455 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:47.403 INFO:tasks.ceph.osd.6.smithi086.stderr:2024-03-25T18:09:47.397+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:09:47.406 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:07.452 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:47.484 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-03-25T18:09:47.477+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:09:47.566 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:07.606 35538== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:47.629 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:07.668 35538== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:47.667 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:07.715 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:47.728 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:07.776 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:47.760 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:00:07.813 35764== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:47.827 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:00:07.883 35763== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:47.899 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:00:07.950 35765== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:47.940 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:00:07.996 35762== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:47.963 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-03-25T18:09:47.957+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:09:47.974 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-03-25T18:09:47.964+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:09:48.122 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:00:08.176 35764== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:48.200 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:00:08.256 35763== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:48.204 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-03-25T18:09:48.195+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:09:48.264 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:00:08.316 35765== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:48.299 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-03-25T18:09:48.293+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:09:48.305 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:00:08.362 35762== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:48.439 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:00:08.493 35764== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:48.447 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:08.492 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:48.467 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:08.522 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:48.517 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:00:08.573 35763== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:48.588 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:00:08.640 35765== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:48.659 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:00:08.715 35762== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:48.700 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:08.739 35538== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:48.706 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:00:08.759 35764== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:48.765 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:08.813 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:48.782 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:00:08.838 35763== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:48.821 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:08.866 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:48.841 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:08.895 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:48.854 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:00:08.905 35765== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:48.953 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:00:09.010 35762== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:48.981 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:00:09.034 35764== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.050 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:00:09.106 35763== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.066 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:09.106 35538== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.118 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:00:09.170 35765== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.130 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:09.178 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.139 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:09.184 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.205 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:09.260 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.250 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:00:09.307 35762== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.254 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:00:09.307 35764== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.314 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:00:09.370 35763== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.382 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:00:09.434 35765== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.382 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:09.422 35538== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.407 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:09.452 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.494 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:09.549 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.497 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:09.545 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.518 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:00:09.572 35764== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.541 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:00:09.598 35762== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.578 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:00:09.634 35763== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.627 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:49.650 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:00:09.702 35765== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.663 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:09.702 35538== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.682 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:09.727 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.717+0000 7fe95eff4640 10 monclient: get_monmap_and_config 2024-03-25T18:09:49.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.717+0000 7fe95eff4640 10 monclient: build_initial_monmap 2024-03-25T18:09:49.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.717+0000 7fe95eff4640 10 monclient: monmap: 2024-03-25T18:09:49.717 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:09:49.718 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:49.718 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:09:49.718125+0000 2024-03-25T18:09:49.718 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:09:49.718125+0000 2024-03-25T18:09:49.718 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:49.718 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:49.718 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:49.718 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:09:49.718 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:09:49.718 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:49.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.717+0000 7fe95eff4640 1 Processor -- start 2024-03-25T18:09:49.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.718+0000 7fe95eff4640 1 -- start start 2024-03-25T18:09:49.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.718+0000 7fe95eff4640 10 monclient: init 2024-03-25T18:09:49.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.718+0000 7fe95eff4640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:49.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.718+0000 7fe95eff4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:09:49.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.718+0000 7fe95eff4640 10 monclient(hunting): picked mon.noname-c con 0x7fe9580b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:49.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.718+0000 7fe95eff4640 10 monclient(hunting): picked mon.noname-a con 0x7fe9580b1730 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:49.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.718+0000 7fe95eff4640 10 monclient(hunting): picked mon.noname-b con 0x7fe95814be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:49.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.718+0000 7fe95eff4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9580af940 con 0x7fe9580b1730 2024-03-25T18:09:49.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.718+0000 7fe95eff4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9580b0a70 con 0x7fe9580b4d60 2024-03-25T18:09:49.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.718+0000 7fe95eff4640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9580b0d50 con 0x7fe95814be90 2024-03-25T18:09:49.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.718+0000 7fe95eff4640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:49.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.718+0000 7fe95eff4640 10 monclient(hunting): authenticate will time out at 6975.957035s 2024-03-25T18:09:49.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.719+0000 7fe95d591640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe95814be90 0x7fe95814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:56810/0 (socket says 172.21.15.64:56810) 2024-03-25T18:09:49.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.719+0000 7fe95d591640 1 -- 172.21.15.64:0/3935905712 learned_addr learned my addr 172.21.15.64:0/3935905712 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:09:49.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.721+0000 7fe9577fe640 1 -- 172.21.15.64:0/3935905712 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061013021 0 0) 0x7fe9580b0d50 con 0x7fe95814be90 2024-03-25T18:09:49.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.721+0000 7fe9577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:49.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.721+0000 7fe9577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:49.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.721+0000 7fe9577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:49.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.721+0000 7fe9577fe640 10 monclient(hunting): my global_id is 4205 2024-03-25T18:09:49.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.721+0000 7fe9577fe640 1 -- 172.21.15.64:0/3935905712 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe948003520 con 0x7fe95814be90 2024-03-25T18:09:49.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.721+0000 7fe9577fe640 1 -- 172.21.15.64:0/3935905712 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036550045 0 0) 0x7fe9580af940 con 0x7fe9580b1730 2024-03-25T18:09:49.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.721+0000 7fe9577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:49.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.721+0000 7fe9577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:49.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.721+0000 7fe9577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:49.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.721+0000 7fe9577fe640 10 monclient(hunting): my global_id is 4206 2024-03-25T18:09:49.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.721+0000 7fe9577fe640 1 -- 172.21.15.64:0/3935905712 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9580b0d50 con 0x7fe9580b1730 2024-03-25T18:09:49.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.721+0000 7fe9577fe640 1 -- 172.21.15.64:0/3935905712 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549158443 0 0) 0x7fe9580b0a70 con 0x7fe9580b4d60 2024-03-25T18:09:49.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.721+0000 7fe9577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:49.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.721+0000 7fe9577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:49.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.721+0000 7fe9577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:49.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.721+0000 7fe9577fe640 10 monclient(hunting): my global_id is 4195 2024-03-25T18:09:49.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.721+0000 7fe9577fe640 1 -- 172.21.15.64:0/3935905712 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9580af940 con 0x7fe9580b4d60 2024-03-25T18:09:49.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.724+0000 7fe9577fe640 1 -- 172.21.15.64:0/3935905712 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3350764257 0 0) 0x7fe948003520 con 0x7fe95814be90 2024-03-25T18:09:49.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.724+0000 7fe9577fe640 1 -- 172.21.15.64:0/3935905712 >> v1:172.21.15.86:6789/0 conn(0x7fe9580b4d60 legacy=0x7fe95813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:49.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.724+0000 7fe9577fe640 1 -- 172.21.15.64:0/3935905712 >> v1:172.21.15.64:6789/0 conn(0x7fe9580b1730 legacy=0x7fe9580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:49.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.724+0000 7fe9577fe640 10 monclient: _finish_hunting 0 2024-03-25T18:09:49.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.724+0000 7fe9577fe640 1 monclient: found mon.noname-b 2024-03-25T18:09:49.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.724+0000 7fe9577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:49.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.724+0000 7fe9577fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:09:49.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.724+0000 7fe9577fe640 1 -- 172.21.15.64:0/3935905712 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9580b1030 con 0x7fe95814be90 2024-03-25T18:09:49.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.724+0000 7fe9577fe640 10 monclient: _finish_auth 0 2024-03-25T18:09:49.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.724+0000 7fe9577fe640 10 monclient: _check_auth_tickets 2024-03-25T18:09:49.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.724+0000 7fe9577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:49.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.724+0000 7fe9577fe640 1 -- 172.21.15.64:0/3935905712 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe950001a10 con 0x7fe95814be90 2024-03-25T18:09:49.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.724+0000 7fe9577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:49.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.724+0000 7fe9577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:49.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.724+0000 7fe9577fe640 10 monclient: dump: 2024-03-25T18:09:49.726 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:09:49.726 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:49.726 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:49.726 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:49.726 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:49.726 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:49.726 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:49.726 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:49.727 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:49.727 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:49.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.724+0000 7fe95eff4640 5 monclient: authenticate success, global_id 4205 2024-03-25T18:09:49.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.724+0000 7fe95eff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:49.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.726+0000 7fe9577fe640 1 -- 172.21.15.64:0/3935905712 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe950001b70 con 0x7fe95814be90 2024-03-25T18:09:49.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.726+0000 7fe9577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:49.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.726+0000 7fe9577fe640 1 -- 172.21.15.64:0/3935905712 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe950001f80 con 0x7fe95814be90 2024-03-25T18:09:49.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.726+0000 7fe9577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:49.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.726+0000 7fe9577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:49.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.726+0000 7fe9577fe640 10 monclient: dump: 2024-03-25T18:09:49.729 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:09:49.729 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:49.729 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:49.729 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:49.729 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:49.729 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:49.729 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:49.729 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:49.729 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:49.729 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:49.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.726+0000 7fe95eff4640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:49.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.726+0000 7fe95eff4640 4 set_mon_vals no callback set 2024-03-25T18:09:49.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.727+0000 7fe95eff4640 10 monclient: shutdown 2024-03-25T18:09:49.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.727+0000 7fe95eff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:49.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.727+0000 7fe95eff4640 1 -- 172.21.15.64:0/3935905712 >> v1:172.21.15.64:6790/0 conn(0x7fe95814be90 legacy=0x7fe95814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:49.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.727+0000 7fe95eff4640 1 -- 172.21.15.64:0/3935905712 shutdown_connections 2024-03-25T18:09:49.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.727+0000 7fe95eff4640 1 -- 172.21.15.64:0/3935905712 >> 172.21.15.64:0/3935905712 conn(0x7fe9580617a0 msgr2=0x7fe958061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:49.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.727+0000 7fe95eff4640 1 -- 172.21.15.64:0/3935905712 shutdown_connections 2024-03-25T18:09:49.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.727+0000 7fe95eff4640 1 -- 172.21.15.64:0/3935905712 wait complete. 2024-03-25T18:09:49.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.727+0000 7fe95eff4640 10 monclient: build_initial_monmap 2024-03-25T18:09:49.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.727+0000 7fe95eff4640 10 monclient: monmap: 2024-03-25T18:09:49.730 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:09:49.730 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:49.730 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:09:49.730 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:09:49.730 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:49.730 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:09:49.730 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:49.730 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:09:49.731 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:09:49.731 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:49.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.728+0000 7fe95eff4640 1 librados: starting msgr at 2024-03-25T18:09:49.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.728+0000 7fe95eff4640 1 librados: starting objecter 2024-03-25T18:09:49.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.728+0000 7fe95eff4640 1 Processor -- start 2024-03-25T18:09:49.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.728+0000 7fe95eff4640 1 -- start start 2024-03-25T18:09:49.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.728+0000 7fe95eff4640 1 librados: setting wanted keys 2024-03-25T18:09:49.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.728+0000 7fe95eff4640 1 librados: calling monclient init 2024-03-25T18:09:49.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.728+0000 7fe95eff4640 10 monclient: init 2024-03-25T18:09:49.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.729+0000 7fe95eff4640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:49.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.729+0000 7fe95eff4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:09:49.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.729+0000 7fe95eff4640 10 monclient(hunting): picked mon.noname-c con 0x7fe9580b1730 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:49.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.729+0000 7fe95eff4640 10 monclient(hunting): picked mon.noname-b con 0x7fe9580b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:49.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.729+0000 7fe95eff4640 10 monclient(hunting): picked mon.noname-a con 0x7fe95814be90 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:49.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.729+0000 7fe95eff4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe958137b10 con 0x7fe95814be90 2024-03-25T18:09:49.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.729+0000 7fe95eff4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9581377e0 con 0x7fe9580b4d60 2024-03-25T18:09:49.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.729+0000 7fe95eff4640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe958137fa0 con 0x7fe9580b1730 2024-03-25T18:09:49.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.729+0000 7fe95eff4640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:49.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.729+0000 7fe95eff4640 10 monclient(hunting): authenticate will time out at 6975.967563s 2024-03-25T18:09:49.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.729+0000 7fe95cd90640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe9580b1730 0x7fe958134eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:56812/0 (socket says 172.21.15.64:56812) 2024-03-25T18:09:49.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.729+0000 7fe95cd90640 1 -- 172.21.15.64:0/287378446 learned_addr learned my addr 172.21.15.64:0/287378446 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:09:49.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.729+0000 7fe9557fa640 10 client.?.objecter ms_handle_connect 0x7fe9580b1730 2024-03-25T18:09:49.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.729+0000 7fe9557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:49.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.729+0000 7fe9557fa640 10 client.?.objecter ms_handle_connect 0x7fe95814be90 2024-03-25T18:09:49.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.729+0000 7fe9557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:49.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.730+0000 7fe9557fa640 10 client.?.objecter ms_handle_connect 0x7fe9580b4d60 2024-03-25T18:09:49.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.730+0000 7fe9557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:49.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.731+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981075211 0 0) 0x7fe958137fa0 con 0x7fe9580b1730 2024-03-25T18:09:49.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.731+0000 7fe9557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:09:49.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.731+0000 7fe9557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:49.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.731+0000 7fe9557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:49.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.731+0000 7fe9557fa640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:09:49.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.731+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe93c004eb0 con 0x7fe9580b1730 2024-03-25T18:09:49.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.731+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484911485 0 0) 0x7fe958137b10 con 0x7fe95814be90 2024-03-25T18:09:49.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.731+0000 7fe9557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:09:49.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.731+0000 7fe9557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:49.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.731+0000 7fe9557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:49.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.731+0000 7fe9557fa640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:09:49.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.731+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe958137fa0 con 0x7fe95814be90 2024-03-25T18:09:49.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.731+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390439287 0 0) 0x7fe9581377e0 con 0x7fe9580b4d60 2024-03-25T18:09:49.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.731+0000 7fe9557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:09:49.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.731+0000 7fe9557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:49.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.731+0000 7fe9557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:49.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.731+0000 7fe9557fa640 10 monclient(hunting): my global_id is 4198 2024-03-25T18:09:49.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.731+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe958137b10 con 0x7fe9580b4d60 2024-03-25T18:09:49.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3942446747 0 0) 0x7fe958137fa0 con 0x7fe95814be90 2024-03-25T18:09:49.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9581377e0 con 0x7fe95814be90 2024-03-25T18:09:49.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe950001e80 con 0x7fe95814be90 2024-03-25T18:09:49.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:49.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:49.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 10 monclient(hunting): dump: 2024-03-25T18:09:49.738 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:09:49.738 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:49.738 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:49.738 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:49.738 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:49.738 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:49.738 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:49.738 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:49.738 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:49.738 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:49.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:49.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 488490965 0 0) 0x7fe93c004eb0 con 0x7fe9580b1730 2024-03-25T18:09:49.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe958137fa0 con 0x7fe9580b1730 2024-03-25T18:09:49.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe94c001990 con 0x7fe9580b1730 2024-03-25T18:09:49.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:49.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:49.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 10 monclient(hunting): dump: 2024-03-25T18:09:49.739 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:09:49.739 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:49.739 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:49.739 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:49.739 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:49.739 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:49.739 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:49.739 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:49.740 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:49.740 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:49.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1242028857 0 0) 0x7fe958137b10 con 0x7fe9580b4d60 2024-03-25T18:09:49.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe93c004eb0 con 0x7fe9580b4d60 2024-03-25T18:09:49.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe9440018b0 con 0x7fe9580b4d60 2024-03-25T18:09:49.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:49.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:49.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.736+0000 7fe9557fa640 10 monclient(hunting): dump: 2024-03-25T18:09:49.740 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:09:49.740 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:49.740 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:49.740 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:49.740 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:49.740 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:49.740 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:49.740 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:49.741 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:49.741 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:49.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3703559687 0 0) 0x7fe958137fa0 con 0x7fe9580b1730 2024-03-25T18:09:49.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 >> v1:172.21.15.86:6789/0 conn(0x7fe9580b4d60 legacy=0x7fe9581daed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:49.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 >> v1:172.21.15.64:6789/0 conn(0x7fe95814be90 legacy=0x7fe9581e3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:49.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe9557fa640 10 monclient: _finish_hunting 0 2024-03-25T18:09:49.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe9557fa640 1 monclient: found mon.c 2024-03-25T18:09:49.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe9557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:49.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe9557fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:49.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe958142030 con 0x7fe9580b1730 2024-03-25T18:09:49.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe9557fa640 10 monclient: _finish_auth 0 2024-03-25T18:09:49.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe9557fa640 10 monclient: _check_auth_tickets 2024-03-25T18:09:49.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe9557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:49.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe95eff4640 5 monclient: authenticate success, global_id 4208 2024-03-25T18:09:49.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe95eff4640 10 monclient: _renew_subs 2024-03-25T18:09:49.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe95eff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:49.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe95eff4640 1 -- 172.21.15.64:0/287378446 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe958140840 con 0x7fe9580b1730 2024-03-25T18:09:49.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe95eff4640 10 client.4208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:49.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe95eff4640 10 monclient: _renew_subs 2024-03-25T18:09:49.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe95eff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:49.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe95eff4640 1 -- 172.21.15.64:0/287378446 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9581357a0 con 0x7fe9580b1730 2024-03-25T18:09:49.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.739+0000 7fe95eff4640 1 librados: init done 2024-03-25T18:09:49.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.740+0000 7fe95eff4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:49.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.740+0000 7fe95eff4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:49.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.740+0000 7fe95eff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:49.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:49.740+0000 7fe95eff4640 1 -- 172.21.15.64:0/287378446 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9580627b0 con 0x7fe9580b1730 2024-03-25T18:09:49.789 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:09.844 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.790 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:09.838 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.790 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:00:09.843 35764== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.828 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:00:09.885 35762== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.866 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:00:09.922 35763== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.915 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:00:09.967 35765== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.927 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:09.966 35538== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:49.949 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:09.994 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.008+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe94c003610 con 0x7fe9580b1730 2024-03-25T18:09:50.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.008+0000 7fe9557fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:50.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.008+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe94c003a20 con 0x7fe9580b1730 2024-03-25T18:09:50.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.008+0000 7fe9557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:50.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.008+0000 7fe9557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:50.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.008+0000 7fe9557fa640 10 monclient: dump: 2024-03-25T18:09:50.009 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:09:50.009 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:50.009 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:50.009 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:50.009 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:50.009 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:50.010 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:50.010 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:50.010 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:50.010 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:50.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.008+0000 7fe956ffd640 4 set_mon_vals no callback set 2024-03-25T18:09:50.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.009+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fe94c003c40 con 0x7fe9580b1730 2024-03-25T18:09:50.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.009+0000 7fe957fff640 10 monclient: get_auth_request con 0x7fe93c06e160 auth_method 0 2024-03-25T18:09:50.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.009+0000 7fe9557fa640 10 client.4208.objecter ms_handle_connect 0x7fe93c06e160 2024-03-25T18:09:50.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.015+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fe94c0547d0 con 0x7fe9580b1730 2024-03-25T18:09:50.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.015+0000 7fe9557fa640 10 client.4208.objecter ms_dispatch 0x7fe958000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:50.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.015+0000 7fe9557fa640 3 client.4208.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:50.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.015+0000 7fe9557fa640 3 client.4208.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:50.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.015+0000 7fe9557fa640 20 client.4208.objecter dump_active .. 0 homeless 2024-03-25T18:09:50.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.015+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe94c0502e0 con 0x7fe9580b1730 2024-03-25T18:09:50.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.015+0000 7fe9557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:50.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.015+0000 7fe9557fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:50.058 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:00:10.111 35764== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.086 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:10.141 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.093 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:00:10.150 35762== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.114 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:10.162 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.131 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:00:10.187 35763== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.180 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:00:10.231 35765== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.198+0000 7fe95eff4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:50.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.198+0000 7fe95eff4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:50.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.198+0000 7fe95eff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:09:50.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.198+0000 7fe95eff4640 1 -- 172.21.15.64:0/287378446 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9580b75b0 con 0x7fe9580b1730 2024-03-25T18:09:50.200 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:10.239 35538== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.215+0000 7fe9557fa640 1 -- 172.21.15.64:0/287378446 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fe94c052210 con 0x7fe9580b1730 2024-03-25T18:09:50.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.215+0000 7fe9557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:50.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.215+0000 7fe9557fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:50.216 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:09:50.216 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:50.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.217+0000 7fe95eff4640 10 librados: watch_flush enter 2024-03-25T18:09:50.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.217+0000 7fe95eff4640 10 librados: watch_flush exit 2024-03-25T18:09:50.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.217+0000 7fe95eff4640 20 client.4208.objecter shutdown clearing up homeless session... 2024-03-25T18:09:50.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.217+0000 7fe95eff4640 10 client.4208.objecter successfully canceled tick 2024-03-25T18:09:50.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.217+0000 7fe95eff4640 1 -- 172.21.15.64:0/287378446 >> v1:172.21.15.64:6800/2915543638 conn(0x7fe93c06e160 legacy=0x7fe93c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:50.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.217+0000 7fe95eff4640 10 monclient: shutdown 2024-03-25T18:09:50.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.217+0000 7fe95eff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:50.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.217+0000 7fe95eff4640 1 -- 172.21.15.64:0/287378446 >> v1:172.21.15.64:6790/0 conn(0x7fe9580b1730 legacy=0x7fe958134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:50.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.217+0000 7fe95eff4640 1 -- 172.21.15.64:0/287378446 shutdown_connections 2024-03-25T18:09:50.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.217+0000 7fe95eff4640 1 -- 172.21.15.64:0/287378446 >> 172.21.15.64:0/287378446 conn(0x7fe9580617a0 msgr2=0x7fe95813c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:50.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.217+0000 7fe95eff4640 1 -- 172.21.15.64:0/287378446 shutdown_connections 2024-03-25T18:09:50.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.217+0000 7fe95eff4640 1 -- 172.21.15.64:0/287378446 wait complete. 2024-03-25T18:09:50.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:50.217+0000 7fe95eff4640 1 librados: shutdown 2024-03-25T18:09:50.219 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:10.263 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.227 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:50.375 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:10.430 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.393 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:00:10.449 35762== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.406 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:10.454 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.465 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:10.505 35538== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.490 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:10.535 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.535 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:00:10.589 35764== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.590 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:00:10.643 35764== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.607 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:00:10.663 35763== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.651 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:00:10.703 35765== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.661 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:00:10.717 35763== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.664 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:10.719 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.693 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:10.741 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.705 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:00:10.757 35765== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.734 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:10.773 35538== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.763 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:10.808 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.937 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:00:10.993 35762== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.957 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:11.012 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.983 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:11.031 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:50.993 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:00:11.049 35762== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:51.006 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:11.045 35538== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:51.240 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:11.285 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:51.255 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:11.303 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:51.296 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:11.341 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:51.493 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:11.533 35538== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:51.526 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:11.580 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:51.550 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:11.590 35538== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:51.582 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:11.636 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:51.854 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:11.902 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:51.910 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:11.958 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:55.862 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:00:15.915 35764== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:55.979 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:00:16.035 35763== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:56.172 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:00:16.223 35765== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:56.530 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:00:16.586 35762== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:57.033 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:57.096 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:17.135 35538== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:57.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.176+0000 7f04a75a4640 10 monclient: get_monmap_and_config 2024-03-25T18:09:57.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.176+0000 7f04a75a4640 10 monclient: build_initial_monmap 2024-03-25T18:09:57.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.176+0000 7f04a75a4640 10 monclient: monmap: 2024-03-25T18:09:57.177 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:09:57.177 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:57.177 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:09:57.177497+0000 2024-03-25T18:09:57.177 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:09:57.177497+0000 2024-03-25T18:09:57.177 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:57.178 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:57.178 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:57.178 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:09:57.178 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:09:57.178 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:57.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.177+0000 7f04a75a4640 1 Processor -- start 2024-03-25T18:09:57.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.177+0000 7f04a75a4640 1 -- start start 2024-03-25T18:09:57.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.177+0000 7f04a75a4640 10 monclient: init 2024-03-25T18:09:57.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.178+0000 7f04a75a4640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:57.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.178+0000 7f04a75a4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:57.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.178+0000 7f04a75a4640 10 monclient(hunting): picked mon.noname-c con 0x7f04a00b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:57.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.178+0000 7f04a75a4640 10 monclient(hunting): picked mon.noname-b con 0x7f04a00b1730 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:57.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.178+0000 7f04a75a4640 10 monclient(hunting): picked mon.noname-a con 0x7f04a014be90 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:57.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.178+0000 7f04a75a4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a00af940 con 0x7f04a014be90 2024-03-25T18:09:57.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.178+0000 7f04a75a4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a00b0a70 con 0x7f04a00b4d60 2024-03-25T18:09:57.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.178+0000 7f04a75a4640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a00b0d50 con 0x7f04a00b1730 2024-03-25T18:09:57.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.178+0000 7f04a75a4640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:57.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.178+0000 7f04a75a4640 10 monclient(hunting): authenticate will time out at 6983.417313s 2024-03-25T18:09:57.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.179+0000 7f04a4b3f640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f04a00b1730 0x7f04a00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:36428/0 (socket says 172.21.15.64:36428) 2024-03-25T18:09:57.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.179+0000 7f04a4b3f640 1 -- 172.21.15.64:0/2394390040 learned_addr learned my addr 172.21.15.64:0/2394390040 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:09:57.184 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.183+0000 7f0497fff640 1 -- 172.21.15.64:0/2394390040 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164834609 0 0) 0x7f04a00b0a70 con 0x7f04a00b4d60 2024-03-25T18:09:57.184 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.183+0000 7f0497fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:57.184 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.183+0000 7f0497fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:57.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.183+0000 7f0497fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:57.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.183+0000 7f0497fff640 10 monclient(hunting): my global_id is 4201 2024-03-25T18:09:57.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.183+0000 7f0497fff640 1 -- 172.21.15.64:0/2394390040 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f048c003520 con 0x7f04a00b4d60 2024-03-25T18:09:57.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.183+0000 7f0497fff640 1 -- 172.21.15.64:0/2394390040 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485560725 0 0) 0x7f04a00b0d50 con 0x7f04a00b1730 2024-03-25T18:09:57.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.183+0000 7f0497fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:57.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.183+0000 7f0497fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:57.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.183+0000 7f0497fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:57.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.183+0000 7f0497fff640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:09:57.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.183+0000 7f0497fff640 1 -- 172.21.15.64:0/2394390040 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a00b0a70 con 0x7f04a00b1730 2024-03-25T18:09:57.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.183+0000 7f0497fff640 1 -- 172.21.15.64:0/2394390040 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550853375 0 0) 0x7f04a00af940 con 0x7f04a014be90 2024-03-25T18:09:57.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.183+0000 7f0497fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:57.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.183+0000 7f0497fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:57.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.183+0000 7f0497fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:57.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.183+0000 7f0497fff640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:09:57.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.183+0000 7f0497fff640 1 -- 172.21.15.64:0/2394390040 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a00b0d50 con 0x7f04a014be90 2024-03-25T18:09:57.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.187+0000 7f0497fff640 1 -- 172.21.15.64:0/2394390040 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 277310148 0 0) 0x7f048c003520 con 0x7f04a00b4d60 2024-03-25T18:09:57.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.187+0000 7f0497fff640 1 -- 172.21.15.64:0/2394390040 >> v1:172.21.15.64:6790/0 conn(0x7f04a00b1730 legacy=0x7f04a00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:57.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.187+0000 7f0497fff640 1 -- 172.21.15.64:0/2394390040 >> v1:172.21.15.64:6789/0 conn(0x7f04a014be90 legacy=0x7f04a014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:57.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.187+0000 7f0497fff640 10 monclient: _finish_hunting 0 2024-03-25T18:09:57.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.187+0000 7f0497fff640 1 monclient: found mon.noname-c 2024-03-25T18:09:57.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.187+0000 7f0497fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:57.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.187+0000 7f0497fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:09:57.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.187+0000 7f0497fff640 1 -- 172.21.15.64:0/2394390040 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a00b1030 con 0x7f04a00b4d60 2024-03-25T18:09:57.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.187+0000 7f0497fff640 10 monclient: _finish_auth 0 2024-03-25T18:09:57.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.187+0000 7f0497fff640 10 monclient: _check_auth_tickets 2024-03-25T18:09:57.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.187+0000 7f0497fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:57.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.188+0000 7f0497fff640 1 -- 172.21.15.64:0/2394390040 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f049c001690 con 0x7f04a00b4d60 2024-03-25T18:09:57.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.188+0000 7f0497fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:57.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.188+0000 7f0497fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:57.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.188+0000 7f0497fff640 10 monclient: dump: 2024-03-25T18:09:57.190 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:09:57.190 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:57.190 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:57.190 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:57.190 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:57.190 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:57.190 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:57.191 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:57.191 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:57.191 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:57.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.189+0000 7f04a75a4640 5 monclient: authenticate success, global_id 4201 2024-03-25T18:09:57.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.189+0000 7f04a75a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:57.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.190+0000 7f0497fff640 1 -- 172.21.15.64:0/2394390040 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f049c001870 con 0x7f04a00b4d60 2024-03-25T18:09:57.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.190+0000 7f0497fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:57.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.190+0000 7f04a75a4640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:57.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.190+0000 7f04a75a4640 4 set_mon_vals no callback set 2024-03-25T18:09:57.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.190+0000 7f0497fff640 1 -- 172.21.15.64:0/2394390040 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f049c001690 con 0x7f04a00b4d60 2024-03-25T18:09:57.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.190+0000 7f0497fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:57.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.190+0000 7f0497fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:57.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.190+0000 7f0497fff640 10 monclient: dump: 2024-03-25T18:09:57.193 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:09:57.193 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:57.194 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:57.194 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:57.194 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:57.194 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:57.194 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:57.194 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:57.194 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:57.194 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:57.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.191+0000 7f04a75a4640 10 monclient: shutdown 2024-03-25T18:09:57.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.191+0000 7f04a75a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:57.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.191+0000 7f04a75a4640 1 -- 172.21.15.64:0/2394390040 >> v1:172.21.15.86:6789/0 conn(0x7f04a00b4d60 legacy=0x7f04a013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:57.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.191+0000 7f04a75a4640 1 -- 172.21.15.64:0/2394390040 shutdown_connections 2024-03-25T18:09:57.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.191+0000 7f04a75a4640 1 -- 172.21.15.64:0/2394390040 >> 172.21.15.64:0/2394390040 conn(0x7f04a00617a0 msgr2=0x7f04a0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:57.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.191+0000 7f04a75a4640 1 -- 172.21.15.64:0/2394390040 shutdown_connections 2024-03-25T18:09:57.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.191+0000 7f04a75a4640 1 -- 172.21.15.64:0/2394390040 wait complete. 2024-03-25T18:09:57.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.192+0000 7f04a75a4640 10 monclient: build_initial_monmap 2024-03-25T18:09:57.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.192+0000 7f04a75a4640 10 monclient: monmap: 2024-03-25T18:09:57.195 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:09:57.195 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:57.195 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:09:57.195 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:09:57.195 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:57.195 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:09:57.195 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:09:57.195 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:09:57.195 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:09:57.196 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:57.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.193+0000 7f04a75a4640 1 librados: starting msgr at 2024-03-25T18:09:57.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.193+0000 7f04a75a4640 1 librados: starting objecter 2024-03-25T18:09:57.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.193+0000 7f04a75a4640 1 Processor -- start 2024-03-25T18:09:57.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.193+0000 7f04a75a4640 1 -- start start 2024-03-25T18:09:57.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.193+0000 7f04a75a4640 1 librados: setting wanted keys 2024-03-25T18:09:57.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.193+0000 7f04a75a4640 1 librados: calling monclient init 2024-03-25T18:09:57.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.193+0000 7f04a75a4640 10 monclient: init 2024-03-25T18:09:57.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.194+0000 7f04a75a4640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:57.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.194+0000 7f04a75a4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:09:57.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.194+0000 7f04a75a4640 10 monclient(hunting): picked mon.noname-b con 0x7f04a00b1730 addr v1:172.21.15.86:6789/0 2024-03-25T18:09:57.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.194+0000 7f04a75a4640 10 monclient(hunting): picked mon.noname-a con 0x7f04a00b4d60 addr v1:172.21.15.64:6789/0 2024-03-25T18:09:57.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.194+0000 7f04a75a4640 10 monclient(hunting): picked mon.noname-c con 0x7f04a014be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:09:57.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.195+0000 7f04a75a4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a01429f0 con 0x7f04a00b4d60 2024-03-25T18:09:57.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.195+0000 7f04a75a4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a0134f90 con 0x7f04a00b1730 2024-03-25T18:09:57.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.195+0000 7f04a4b3f640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f04a00b4d60 0x7f04a0134170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:34380/0 (socket says 172.21.15.64:34380) 2024-03-25T18:09:57.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.195+0000 7f04a4b3f640 1 -- 172.21.15.64:0/2364694245 learned_addr learned my addr 172.21.15.64:0/2364694245 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:09:57.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.195+0000 7f04a75a4640 1 -- 172.21.15.64:0/2364694245 --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a0139f10 con 0x7f04a014be90 2024-03-25T18:09:57.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.195+0000 7f04a75a4640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:57.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.195+0000 7f04a75a4640 10 monclient(hunting): authenticate will time out at 6983.433957s 2024-03-25T18:09:57.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.195+0000 7f0495ffb640 10 client.?.objecter ms_handle_connect 0x7f04a00b4d60 2024-03-25T18:09:57.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.195+0000 7f0495ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:57.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.195+0000 7f0495ffb640 10 client.?.objecter ms_handle_connect 0x7f04a00b1730 2024-03-25T18:09:57.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.195+0000 7f0495ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:57.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.196+0000 7f0495ffb640 10 client.?.objecter ms_handle_connect 0x7f04a014be90 2024-03-25T18:09:57.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.196+0000 7f0495ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:57.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.198+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801530043 0 0) 0x7f04a0139f10 con 0x7f04a014be90 2024-03-25T18:09:57.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.198+0000 7f0495ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:57.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.198+0000 7f0495ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:57.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.198+0000 7f0495ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:57.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.198+0000 7f0495ffb640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:09:57.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.198+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0484004eb0 con 0x7f04a014be90 2024-03-25T18:09:57.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.199+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578638159 0 0) 0x7f04a01429f0 con 0x7f04a00b4d60 2024-03-25T18:09:57.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.199+0000 7f0495ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:57.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.199+0000 7f0495ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:57.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.199+0000 7f0495ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:57.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.199+0000 7f0495ffb640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:09:57.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.199+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a0139f10 con 0x7f04a00b4d60 2024-03-25T18:09:57.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.199+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99472315 0 0) 0x7f04a0134f90 con 0x7f04a00b1730 2024-03-25T18:09:57.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.199+0000 7f0495ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:57.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.199+0000 7f0495ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:57.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.199+0000 7f0495ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:57.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.199+0000 7f0495ffb640 10 monclient(hunting): my global_id is 4204 2024-03-25T18:09:57.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.199+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a01429f0 con 0x7f04a00b1730 2024-03-25T18:09:57.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.205+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1230553794 0 0) 0x7f04a01429f0 con 0x7f04a00b1730 2024-03-25T18:09:57.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.205+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04a0134f90 con 0x7f04a00b1730 2024-03-25T18:09:57.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.205+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f049c001690 con 0x7f04a00b1730 2024-03-25T18:09:57.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.205+0000 7f0495ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:57.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.205+0000 7f0495ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:57.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.205+0000 7f0495ffb640 10 monclient(hunting): dump: 2024-03-25T18:09:57.206 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:09:57.206 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:57.206 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:57.206 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:57.207 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:57.207 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:57.207 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:57.207 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:57.207 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:57.207 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:57.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.205+0000 7f0495ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:57.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.206+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 91470587 0 0) 0x7f04a0139f10 con 0x7f04a00b4d60 2024-03-25T18:09:57.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.206+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04a01429f0 con 0x7f04a00b4d60 2024-03-25T18:09:57.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.207+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0490001970 con 0x7f04a00b4d60 2024-03-25T18:09:57.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.207+0000 7f0495ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:57.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.207+0000 7f0495ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:57.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.207+0000 7f0495ffb640 10 monclient(hunting): dump: 2024-03-25T18:09:57.209 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:09:57.209 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:57.209 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:57.209 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:57.209 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:57.209 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:57.209 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:57.209 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:57.209 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:57.209 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:57.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.207+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3980408963 0 0) 0x7f0484004eb0 con 0x7f04a014be90 2024-03-25T18:09:57.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.207+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04a0139f10 con 0x7f04a014be90 2024-03-25T18:09:57.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.207+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0498001bd0 con 0x7f04a014be90 2024-03-25T18:09:57.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.207+0000 7f0495ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:57.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.207+0000 7f0495ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:57.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.207+0000 7f0495ffb640 10 monclient(hunting): dump: 2024-03-25T18:09:57.210 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:09:57.210 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:57.210 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:57.210 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:57.210 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:57.210 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:57.210 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:57.210 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:57.210 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:57.210 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:57.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 687467322 0 0) 0x7f04a0134f90 con 0x7f04a00b1730 2024-03-25T18:09:57.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 >> v1:172.21.15.64:6790/0 conn(0x7f04a014be90 legacy=0x7f04a0134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:57.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 >> v1:172.21.15.64:6789/0 conn(0x7f04a00b4d60 legacy=0x7f04a0134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:57.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f0495ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:09:57.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f0495ffb640 1 monclient: found mon.b 2024-03-25T18:09:57.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f0495ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:57.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f0495ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:57.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a0142e80 con 0x7f04a00b1730 2024-03-25T18:09:57.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f0495ffb640 10 monclient: _finish_auth 0 2024-03-25T18:09:57.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f0495ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:09:57.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f0495ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:57.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f04a75a4640 5 monclient: authenticate success, global_id 4204 2024-03-25T18:09:57.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f04a75a4640 10 monclient: _renew_subs 2024-03-25T18:09:57.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f04a75a4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:57.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f04a75a4640 1 -- 172.21.15.64:0/2364694245 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a01431c0 con 0x7f04a00b1730 2024-03-25T18:09:57.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f04a75a4640 10 client.4204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:57.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f04a75a4640 10 monclient: _renew_subs 2024-03-25T18:09:57.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f04a75a4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:57.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f04a75a4640 1 -- 172.21.15.64:0/2364694245 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a0139140 con 0x7f04a00b1730 2024-03-25T18:09:57.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.209+0000 7f04a75a4640 1 librados: init done 2024-03-25T18:09:57.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.211+0000 7f04a75a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:57.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.211+0000 7f04a75a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:57.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.211+0000 7f04a75a4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:57.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.211+0000 7f04a75a4640 1 -- 172.21.15.64:0/2364694245 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a013a430 con 0x7f04a00b1730 2024-03-25T18:09:57.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.232+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f049c001140 con 0x7f04a00b1730 2024-03-25T18:09:57.233 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.232+0000 7f0495ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:57.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.232+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f049c003950 con 0x7f04a00b1730 2024-03-25T18:09:57.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.232+0000 7f0495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:57.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.232+0000 7f0495ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:57.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.232+0000 7f0495ffb640 10 monclient: dump: 2024-03-25T18:09:57.234 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:09:57.234 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:09:57.234 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:57.234 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:09:57.234 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:57.234 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:09:57.234 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:09:57.234 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:09:57.234 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:09:57.234 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:09:57.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.232+0000 7f04977fe640 4 set_mon_vals no callback set 2024-03-25T18:09:57.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.233+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f049c019920 con 0x7f04a00b1730 2024-03-25T18:09:57.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.234+0000 7f04a4b3f640 10 monclient: get_auth_request con 0x7f048406e0f0 auth_method 0 2024-03-25T18:09:57.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.234+0000 7f0495ffb640 10 client.4204.objecter ms_handle_connect 0x7f048406e0f0 2024-03-25T18:09:57.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.237+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f049c053470 con 0x7f04a00b1730 2024-03-25T18:09:57.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.237+0000 7f0495ffb640 10 client.4204.objecter ms_dispatch 0x7f04a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:57.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.237+0000 7f0495ffb640 3 client.4204.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:57.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.237+0000 7f0495ffb640 3 client.4204.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:57.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.237+0000 7f0495ffb640 20 client.4204.objecter dump_active .. 0 homeless 2024-03-25T18:09:57.285 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:17.330 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:57.416 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:17.470 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:57.479 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:17.526 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:09:57.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.569+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f049c04ef80 con 0x7f04a00b1730 2024-03-25T18:09:57.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.569+0000 7f0495ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:57.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.569+0000 7f0495ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:57.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.871+0000 7f04a75a4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:57.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.871+0000 7f04a75a4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:57.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.871+0000 7f04a75a4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:09:57.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.871+0000 7f04a75a4640 1 -- 172.21.15.64:0/2364694245 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04a00afbd0 con 0x7f04a00b1730 2024-03-25T18:09:57.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.931+0000 7f0495ffb640 1 -- 172.21.15.64:0/2364694245 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f049c050eb0 con 0x7f04a00b1730 2024-03-25T18:09:57.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.931+0000 7f0495ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:57.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.931+0000 7f0495ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:57.932 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:09:57.932 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:57.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.934+0000 7f04a75a4640 10 librados: watch_flush enter 2024-03-25T18:09:57.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.934+0000 7f04a75a4640 10 librados: watch_flush exit 2024-03-25T18:09:57.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.934+0000 7f04a75a4640 20 client.4204.objecter shutdown clearing up homeless session... 2024-03-25T18:09:57.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.934+0000 7f04a75a4640 10 client.4204.objecter successfully canceled tick 2024-03-25T18:09:57.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.934+0000 7f04a75a4640 1 -- 172.21.15.64:0/2364694245 >> v1:172.21.15.64:6800/2915543638 conn(0x7f048406e0f0 legacy=0x7f04840705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:57.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.934+0000 7f04a75a4640 10 monclient: shutdown 2024-03-25T18:09:57.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.934+0000 7f04a75a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:57.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.934+0000 7f04a75a4640 1 -- 172.21.15.64:0/2364694245 >> v1:172.21.15.86:6789/0 conn(0x7f04a00b1730 legacy=0x7f04a0140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:57.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.934+0000 7f04a75a4640 1 -- 172.21.15.64:0/2364694245 shutdown_connections 2024-03-25T18:09:57.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.934+0000 7f04a75a4640 1 -- 172.21.15.64:0/2364694245 >> 172.21.15.64:0/2364694245 conn(0x7f04a00617a0 msgr2=0x7f04a00b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:57.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.934+0000 7f04a75a4640 1 -- 172.21.15.64:0/2364694245 shutdown_connections 2024-03-25T18:09:57.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.934+0000 7f04a75a4640 1 -- 172.21.15.64:0/2364694245 wait complete. 2024-03-25T18:09:57.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:09:57.934+0000 7f04a75a4640 1 librados: shutdown 2024-03-25T18:09:57.947 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:01.559 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:00:21.612 35764== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:01.565 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:00:21.618 35764== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:01.823 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:00:21.879 35763== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:01.827 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:00:21.883 35763== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:01.958 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:00:22.009 35765== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:01.962 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:00:22.013 35765== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:02.373 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:00:22.429 35762== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:02.377 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:00:22.433 35762== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:02.591 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:22.630 35538== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:02.595 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:22.634 35538== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:02.878 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:22.923 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:02.881 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:22.927 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:03.121 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:23.169 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:03.126 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:23.174 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:03.315 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:23.370 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:03.320 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:23.374 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:04.754 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:04.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.847+0000 7f77d9be8640 10 monclient: get_monmap_and_config 2024-03-25T18:10:04.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.847+0000 7f77d9be8640 10 monclient: build_initial_monmap 2024-03-25T18:10:04.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.848+0000 7f77d9be8640 10 monclient: monmap: 2024-03-25T18:10:04.848 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:10:04.848 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:04.849 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:10:04.848877+0000 2024-03-25T18:10:04.849 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:10:04.848877+0000 2024-03-25T18:10:04.849 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:04.849 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:04.849 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:10:04.849 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:10:04.849 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:10:04.849 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:04.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.848+0000 7f77d9be8640 1 Processor -- start 2024-03-25T18:10:04.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.848+0000 7f77d9be8640 1 -- start start 2024-03-25T18:10:04.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.848+0000 7f77d9be8640 10 monclient: init 2024-03-25T18:10:04.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.849+0000 7f77d9be8640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:04.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.849+0000 7f77d9be8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:10:04.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.849+0000 7f77d9be8640 10 monclient(hunting): picked mon.noname-b con 0x7f77d40b4d60 addr v1:172.21.15.64:6790/0 2024-03-25T18:10:04.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.849+0000 7f77d9be8640 10 monclient(hunting): picked mon.noname-a con 0x7f77d40b1730 addr v1:172.21.15.64:6789/0 2024-03-25T18:10:04.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.849+0000 7f77d9be8640 10 monclient(hunting): picked mon.noname-c con 0x7f77d414be90 addr v1:172.21.15.86:6789/0 2024-03-25T18:10:04.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.849+0000 7f77d9be8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77d40af940 con 0x7f77d40b1730 2024-03-25T18:10:04.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.849+0000 7f77d9be8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77d40b0a70 con 0x7f77d414be90 2024-03-25T18:10:04.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.849+0000 7f77d9be8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77d40b0d50 con 0x7f77d40b4d60 2024-03-25T18:10:04.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.849+0000 7f77d9be8640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:04.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.849+0000 7f77d9be8640 10 monclient(hunting): authenticate will time out at 6991.088043s 2024-03-25T18:10:04.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.850+0000 7f77d37fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f77d40b4d60 0x7f77d413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:36446/0 (socket says 172.21.15.64:36446) 2024-03-25T18:10:04.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.850+0000 7f77d37fe640 1 -- 172.21.15.64:0/1109653564 learned_addr learned my addr 172.21.15.64:0/1109653564 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:10:04.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.852+0000 7f77d27fc640 1 -- 172.21.15.64:0/1109653564 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221974243 0 0) 0x7f77d40af940 con 0x7f77d40b1730 2024-03-25T18:10:04.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.852+0000 7f77d27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:04.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.852+0000 7f77d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:04.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.852+0000 7f77d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:04.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.852+0000 7f77d27fc640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:10:04.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.852+0000 7f77d27fc640 1 -- 172.21.15.64:0/1109653564 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77bc003520 con 0x7f77d40b1730 2024-03-25T18:10:04.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.853+0000 7f77d27fc640 1 -- 172.21.15.64:0/1109653564 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345460799 0 0) 0x7f77d40b0d50 con 0x7f77d40b4d60 2024-03-25T18:10:04.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.853+0000 7f77d27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:04.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.853+0000 7f77d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:04.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.853+0000 7f77d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:04.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.853+0000 7f77d27fc640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:10:04.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.853+0000 7f77d27fc640 1 -- 172.21.15.64:0/1109653564 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77d40af940 con 0x7f77d40b4d60 2024-03-25T18:10:04.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.854+0000 7f77d27fc640 1 -- 172.21.15.64:0/1109653564 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453705137 0 0) 0x7f77d40b0a70 con 0x7f77d414be90 2024-03-25T18:10:04.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.854+0000 7f77d27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:04.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.854+0000 7f77d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:04.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.854+0000 7f77d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:04.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.854+0000 7f77d27fc640 10 monclient(hunting): my global_id is 4207 2024-03-25T18:10:04.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.854+0000 7f77d27fc640 1 -- 172.21.15.64:0/1109653564 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77d40b0d50 con 0x7f77d414be90 2024-03-25T18:10:04.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.856+0000 7f77d27fc640 1 -- 172.21.15.64:0/1109653564 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 938220592 0 0) 0x7f77bc003520 con 0x7f77d40b1730 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.856+0000 7f77d27fc640 1 -- 172.21.15.64:0/1109653564 >> v1:172.21.15.64:6790/0 conn(0x7f77d40b4d60 legacy=0x7f77d413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.856+0000 7f77d27fc640 1 -- 172.21.15.64:0/1109653564 >> v1:172.21.15.86:6789/0 conn(0x7f77d414be90 legacy=0x7f77d414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.856+0000 7f77d27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.856+0000 7f77d27fc640 1 monclient: found mon.noname-a 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.856+0000 7f77d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.856+0000 7f77d27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.856+0000 7f77d27fc640 1 -- 172.21.15.64:0/1109653564 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d40b1030 con 0x7f77d40b1730 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.856+0000 7f77d27fc640 10 monclient: _finish_auth 0 2024-03-25T18:10:04.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.856+0000 7f77d27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:10:04.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.856+0000 7f77d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:04.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.856+0000 7f77d9be8640 5 monclient: authenticate success, global_id 4218 2024-03-25T18:10:04.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.856+0000 7f77d9be8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:04.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.856+0000 7f77d27fc640 1 -- 172.21.15.64:0/1109653564 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f77c8001580 con 0x7f77d40b1730 2024-03-25T18:10:04.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.856+0000 7f77d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:04.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.856+0000 7f77d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:04.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.856+0000 7f77d27fc640 10 monclient: dump: 2024-03-25T18:10:04.860 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:04.860 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:04.860 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:04.860 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:04.860 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:04.860 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:04.860 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:04.860 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.857+0000 7f77d9be8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.858+0000 7f77d27fc640 1 -- 172.21.15.64:0/1109653564 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77c80017d0 con 0x7f77d40b1730 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.858+0000 7f77d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.858+0000 7f77d9be8640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:04.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.858+0000 7f77d9be8640 4 set_mon_vals no callback set 2024-03-25T18:10:04.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.858+0000 7f77d27fc640 1 -- 172.21.15.64:0/1109653564 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f77c8001b60 con 0x7f77d40b1730 2024-03-25T18:10:04.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.858+0000 7f77d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:04.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.858+0000 7f77d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:04.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.858+0000 7f77d27fc640 10 monclient: dump: 2024-03-25T18:10:04.865 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:04.865 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:04.865 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:04.865 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:04.865 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:04.865 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:04.865 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:04.865 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:04.866 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:04.866 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:04.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.859+0000 7f77d9be8640 10 monclient: shutdown 2024-03-25T18:10:04.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.859+0000 7f77d9be8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:04.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.859+0000 7f77d9be8640 1 -- 172.21.15.64:0/1109653564 >> v1:172.21.15.64:6789/0 conn(0x7f77d40b1730 legacy=0x7f77d40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:04.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.859+0000 7f77d9be8640 1 -- 172.21.15.64:0/1109653564 shutdown_connections 2024-03-25T18:10:04.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.859+0000 7f77d9be8640 1 -- 172.21.15.64:0/1109653564 >> 172.21.15.64:0/1109653564 conn(0x7f77d40617a0 msgr2=0x7f77d4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:04.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.859+0000 7f77d9be8640 1 -- 172.21.15.64:0/1109653564 shutdown_connections 2024-03-25T18:10:04.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.859+0000 7f77d9be8640 1 -- 172.21.15.64:0/1109653564 wait complete. 2024-03-25T18:10:04.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.860+0000 7f77d9be8640 10 monclient: build_initial_monmap 2024-03-25T18:10:04.866 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.860+0000 7f77d9be8640 10 monclient: monmap: 2024-03-25T18:10:04.866 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:10:04.866 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:04.867 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:10:04.867 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:10:04.867 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:04.867 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:10:04.867 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:10:04.867 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:10:04.867 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:10:04.867 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:04.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.861+0000 7f77d9be8640 1 librados: starting msgr at 2024-03-25T18:10:04.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.861+0000 7f77d9be8640 1 librados: starting objecter 2024-03-25T18:10:04.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.861+0000 7f77d9be8640 1 Processor -- start 2024-03-25T18:10:04.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.861+0000 7f77d9be8640 1 -- start start 2024-03-25T18:10:04.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.861+0000 7f77d9be8640 1 librados: setting wanted keys 2024-03-25T18:10:04.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.861+0000 7f77d9be8640 1 librados: calling monclient init 2024-03-25T18:10:04.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.861+0000 7f77d9be8640 10 monclient: init 2024-03-25T18:10:04.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.862+0000 7f77d9be8640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:04.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.862+0000 7f77d9be8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:10:04.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.862+0000 7f77d9be8640 10 monclient(hunting): picked mon.noname-a con 0x7f77d40b1730 addr v1:172.21.15.64:6789/0 2024-03-25T18:10:04.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.862+0000 7f77d9be8640 10 monclient(hunting): picked mon.noname-b con 0x7f77d40b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:10:04.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.862+0000 7f77d9be8640 10 monclient(hunting): picked mon.noname-c con 0x7f77d414be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:10:04.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.863+0000 7f77d9be8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77d41d7690 con 0x7f77d40b1730 2024-03-25T18:10:04.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.863+0000 7f77d9be8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77d4137d30 con 0x7f77d40b4d60 2024-03-25T18:10:04.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.863+0000 7f77d9be8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77d4137fa0 con 0x7f77d414be90 2024-03-25T18:10:04.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.863+0000 7f77d37fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f77d40b1730 0x7f77d412fe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:34392/0 (socket says 172.21.15.64:34392) 2024-03-25T18:10:04.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.863+0000 7f77d2ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f77d40b4d60 0x7f77d41d7b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:38328/0 (socket says 172.21.15.64:38328) 2024-03-25T18:10:04.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.863+0000 7f77d2ffd640 1 -- 172.21.15.64:0/2518165987 learned_addr learned my addr 172.21.15.64:0/2518165987 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:10:04.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.863+0000 7f77d9be8640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:04.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.863+0000 7f77d9be8640 10 monclient(hunting): authenticate will time out at 6991.102097s 2024-03-25T18:10:04.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.863+0000 7f77bbfff640 10 client.?.objecter ms_handle_connect 0x7f77d40b4d60 2024-03-25T18:10:04.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.863+0000 7f77bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:04.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.863+0000 7f77bbfff640 10 client.?.objecter ms_handle_connect 0x7f77d414be90 2024-03-25T18:10:04.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.863+0000 7f77bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:04.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.863+0000 7f77bbfff640 10 client.?.objecter ms_handle_connect 0x7f77d40b1730 2024-03-25T18:10:04.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.863+0000 7f77bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:04.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.866+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564855280 0 0) 0x7f77d4137d30 con 0x7f77d40b4d60 2024-03-25T18:10:04.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.866+0000 7f77bbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:04.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.866+0000 7f77bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:04.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.866+0000 7f77bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:04.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.866+0000 7f77bbfff640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:10:04.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.866+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77c0004eb0 con 0x7f77d40b4d60 2024-03-25T18:10:04.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.866+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412204553 0 0) 0x7f77d4137fa0 con 0x7f77d414be90 2024-03-25T18:10:04.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.866+0000 7f77bbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:04.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.866+0000 7f77bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:04.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.866+0000 7f77bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:04.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.866+0000 7f77bbfff640 10 monclient(hunting): my global_id is 4220 2024-03-25T18:10:04.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.866+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77d4137d30 con 0x7f77d414be90 2024-03-25T18:10:04.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.866+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470494841 0 0) 0x7f77d41d7690 con 0x7f77d40b1730 2024-03-25T18:10:04.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.866+0000 7f77bbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:04.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.866+0000 7f77bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:04.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.866+0000 7f77bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:04.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.866+0000 7f77bbfff640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:10:04.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.866+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77d4137fa0 con 0x7f77d40b1730 2024-03-25T18:10:04.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.872+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 81263611 0 0) 0x7f77c0004eb0 con 0x7f77d40b4d60 2024-03-25T18:10:04.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.872+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77d41d7690 con 0x7f77d40b4d60 2024-03-25T18:10:04.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.872+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f77c8001ae0 con 0x7f77d40b4d60 2024-03-25T18:10:04.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.872+0000 7f77bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:04.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.872+0000 7f77bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:04.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.872+0000 7f77bbfff640 10 monclient(hunting): dump: 2024-03-25T18:10:04.873 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:04.873 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:04.873 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:04.873 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:04.873 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:04.873 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:04.873 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:04.873 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:04.873 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:04.873 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:04.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.872+0000 7f77bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:04.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.873+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3629722473 0 0) 0x7f77d4137d30 con 0x7f77d414be90 2024-03-25T18:10:04.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.873+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77c0004eb0 con 0x7f77d414be90 2024-03-25T18:10:04.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.873+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f77cc001b50 con 0x7f77d414be90 2024-03-25T18:10:04.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.873+0000 7f77bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:04.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.873+0000 7f77bbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:04.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.873+0000 7f77bbfff640 10 monclient(hunting): dump: 2024-03-25T18:10:04.875 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:04.875 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:04.876 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:04.876 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:04.876 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:04.876 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:04.876 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:04.876 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:04.876 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:04.876 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:04.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.874+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4242927374 0 0) 0x7f77d4137fa0 con 0x7f77d40b1730 2024-03-25T18:10:04.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.874+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77d4137d30 con 0x7f77d40b1730 2024-03-25T18:10:04.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.874+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f77c40018b0 con 0x7f77d40b1730 2024-03-25T18:10:04.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.874+0000 7f77bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:04.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.874+0000 7f77bbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:04.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.874+0000 7f77bbfff640 10 monclient(hunting): dump: 2024-03-25T18:10:04.877 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:04.877 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:04.877 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:04.877 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:04.877 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:04.877 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:04.877 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:04.877 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:04.877 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:04.877 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:04.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3819093041 0 0) 0x7f77d41d7690 con 0x7f77d40b4d60 2024-03-25T18:10:04.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 >> v1:172.21.15.64:6790/0 conn(0x7f77d414be90 legacy=0x7f77d41e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:04.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 >> v1:172.21.15.64:6789/0 conn(0x7f77d40b1730 legacy=0x7f77d412fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:04.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77bbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:10:04.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77bbfff640 1 monclient: found mon.b 2024-03-25T18:10:04.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:04.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77bbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:04.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d4143690 con 0x7f77d40b4d60 2024-03-25T18:10:04.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77bbfff640 10 monclient: _finish_auth 0 2024-03-25T18:10:04.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77bbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:10:04.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:04.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77d9be8640 5 monclient: authenticate success, global_id 4210 2024-03-25T18:10:04.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77d9be8640 10 monclient: _renew_subs 2024-03-25T18:10:04.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77d9be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:04.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77d9be8640 1 -- 172.21.15.64:0/2518165987 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d4135c10 con 0x7f77d40b4d60 2024-03-25T18:10:04.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77d9be8640 10 client.4210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:04.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77d9be8640 10 monclient: _renew_subs 2024-03-25T18:10:04.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77d9be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:04.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77d9be8640 1 -- 172.21.15.64:0/2518165987 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d41362b0 con 0x7f77d40b4d60 2024-03-25T18:10:04.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.876+0000 7f77d9be8640 1 librados: init done 2024-03-25T18:10:04.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.878+0000 7f77d9be8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:04.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.878+0000 7f77d9be8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:04.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.878+0000 7f77d9be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:04.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.878+0000 7f77d9be8640 1 -- 172.21.15.64:0/2518165987 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d40627b0 con 0x7f77d40b4d60 2024-03-25T18:10:04.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.899+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77c80038a0 con 0x7f77d40b4d60 2024-03-25T18:10:04.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.899+0000 7f77bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:04.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.899+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f77c8003cb0 con 0x7f77d40b4d60 2024-03-25T18:10:04.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.899+0000 7f77bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:04.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.899+0000 7f77bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:04.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.899+0000 7f77bbfff640 10 monclient: dump: 2024-03-25T18:10:04.900 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:04.900 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:04.900 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:04.900 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:04.900 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:04.900 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:04.901 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:04.901 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:04.901 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:04.901 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:04.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.899+0000 7f77d1ffb640 4 set_mon_vals no callback set 2024-03-25T18:10:04.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.900+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f77c8003ed0 con 0x7f77d40b4d60 2024-03-25T18:10:04.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.900+0000 7f77d37fe640 10 monclient: get_auth_request con 0x7f77c006e2b0 auth_method 0 2024-03-25T18:10:04.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.901+0000 7f77bbfff640 10 client.4210.objecter ms_handle_connect 0x7f77c006e2b0 2024-03-25T18:10:04.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.904+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f77c80548d0 con 0x7f77d40b4d60 2024-03-25T18:10:04.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.904+0000 7f77bbfff640 10 client.4210.objecter ms_dispatch 0x7f77d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:04.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.904+0000 7f77bbfff640 3 client.4210.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:04.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.904+0000 7f77bbfff640 3 client.4210.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:04.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:04.904+0000 7f77bbfff640 20 client.4210.objecter dump_active .. 0 homeless 2024-03-25T18:10:05.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.156+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f77c80503e0 con 0x7f77d40b4d60 2024-03-25T18:10:05.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.156+0000 7f77bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:05.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.156+0000 7f77bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:05.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.381+0000 7f77d9be8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:05.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.381+0000 7f77d9be8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:05.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.381+0000 7f77d9be8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:05.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.381+0000 7f77d9be8640 1 -- 172.21.15.64:0/2518165987 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77d41d7840 con 0x7f77d40b4d60 2024-03-25T18:10:05.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.396+0000 7f77bbfff640 1 -- 172.21.15.64:0/2518165987 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f77d41d7840 con 0x7f77d40b4d60 2024-03-25T18:10:05.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.396+0000 7f77bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:05.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.396+0000 7f77bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:05.397 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:10:05.397 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:05.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.398+0000 7f77d9be8640 10 librados: watch_flush enter 2024-03-25T18:10:05.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.398+0000 7f77d9be8640 10 librados: watch_flush exit 2024-03-25T18:10:05.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.399+0000 7f77d9be8640 20 client.4210.objecter shutdown clearing up homeless session... 2024-03-25T18:10:05.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.399+0000 7f77d9be8640 10 client.4210.objecter successfully canceled tick 2024-03-25T18:10:05.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.399+0000 7f77d9be8640 1 -- 172.21.15.64:0/2518165987 >> v1:172.21.15.64:6800/2915543638 conn(0x7f77c006e2b0 legacy=0x7f77c0070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:05.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.399+0000 7f77d9be8640 10 monclient: shutdown 2024-03-25T18:10:05.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.399+0000 7f77d9be8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:05.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.399+0000 7f77d9be8640 1 -- 172.21.15.64:0/2518165987 >> v1:172.21.15.86:6789/0 conn(0x7f77d40b4d60 legacy=0x7f77d41d7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:05.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.399+0000 7f77d9be8640 1 -- 172.21.15.64:0/2518165987 shutdown_connections 2024-03-25T18:10:05.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.399+0000 7f77d9be8640 1 -- 172.21.15.64:0/2518165987 >> 172.21.15.64:0/2518165987 conn(0x7f77d40617a0 msgr2=0x7f77d413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:05.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.399+0000 7f77d9be8640 1 -- 172.21.15.64:0/2518165987 shutdown_connections 2024-03-25T18:10:05.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.399+0000 7f77d9be8640 1 -- 172.21.15.64:0/2518165987 wait complete. 2024-03-25T18:10:05.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:05.399+0000 7f77d9be8640 1 librados: shutdown 2024-03-25T18:10:05.409 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:05.911 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:00:25.967 35763== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:05.987 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:00:26.040 35764== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:06.350 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:00:26.401 35765== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:06.622 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:00:26.678 35762== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:06.924 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:00:26.963 35538== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:07.225 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:00:27.272 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:07.257 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:00:27.303 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:07.802 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:00:27.856 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:10:12.216 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:12.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.304+0000 7f1538e57640 10 monclient: get_monmap_and_config 2024-03-25T18:10:12.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.304+0000 7f1538e57640 10 monclient: build_initial_monmap 2024-03-25T18:10:12.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.304+0000 7f1538e57640 10 monclient: monmap: 2024-03-25T18:10:12.306 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:10:12.306 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:12.306 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:10:12.306467+0000 2024-03-25T18:10:12.306 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:10:12.306467+0000 2024-03-25T18:10:12.306 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:12.306 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:12.306 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:10:12.307 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:10:12.307 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:10:12.307 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:12.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.305+0000 7f1538e57640 1 Processor -- start 2024-03-25T18:10:12.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.305+0000 7f1538e57640 1 -- start start 2024-03-25T18:10:12.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.305+0000 7f1538e57640 10 monclient: init 2024-03-25T18:10:12.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.306+0000 7f1538e57640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:12.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.306+0000 7f1538e57640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:12.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.306+0000 7f1538e57640 10 monclient(hunting): picked mon.noname-b con 0x7f15340b4d60 addr v1:172.21.15.64:6790/0 2024-03-25T18:10:12.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.306+0000 7f1538e57640 10 monclient(hunting): picked mon.noname-c con 0x7f15340b1730 addr v1:172.21.15.86:6789/0 2024-03-25T18:10:12.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.306+0000 7f1538e57640 10 monclient(hunting): picked mon.noname-a con 0x7f153414be90 addr v1:172.21.15.64:6789/0 2024-03-25T18:10:12.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.306+0000 7f1538e57640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15340af940 con 0x7f153414be90 2024-03-25T18:10:12.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.306+0000 7f1538e57640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15340b0a70 con 0x7f15340b1730 2024-03-25T18:10:12.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.306+0000 7f1538e57640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15340b0d50 con 0x7f15340b4d60 2024-03-25T18:10:12.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.306+0000 7f1538e57640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:12.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.306+0000 7f1538e57640 10 monclient(hunting): authenticate will time out at 6998.545913s 2024-03-25T18:10:12.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.306+0000 7f1532d9d640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f153414be90 0x7f153414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:49376/0 (socket says 172.21.15.64:49376) 2024-03-25T18:10:12.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.306+0000 7f1532d9d640 1 -- 172.21.15.64:0/753281817 learned_addr learned my addr 172.21.15.64:0/753281817 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:10:12.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.309+0000 7f153159a640 1 -- 172.21.15.64:0/753281817 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384952298 0 0) 0x7f15340af940 con 0x7f153414be90 2024-03-25T18:10:12.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.309+0000 7f153159a640 10 monclient(hunting): _negotiate 2024-03-25T18:10:12.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.309+0000 7f153159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:12.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.309+0000 7f153159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:12.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.309+0000 7f153159a640 10 monclient(hunting): my global_id is 4224 2024-03-25T18:10:12.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.309+0000 7f153159a640 1 -- 172.21.15.64:0/753281817 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f151c003520 con 0x7f153414be90 2024-03-25T18:10:12.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.311+0000 7f153159a640 1 -- 172.21.15.64:0/753281817 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402372387 0 0) 0x7f15340b0d50 con 0x7f15340b4d60 2024-03-25T18:10:12.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.311+0000 7f153159a640 10 monclient(hunting): _negotiate 2024-03-25T18:10:12.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.311+0000 7f153159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:12.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.311+0000 7f153159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:12.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.311+0000 7f153159a640 10 monclient(hunting): my global_id is 4223 2024-03-25T18:10:12.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.311+0000 7f153159a640 1 -- 172.21.15.64:0/753281817 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15340af940 con 0x7f15340b4d60 2024-03-25T18:10:12.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.311+0000 7f153159a640 1 -- 172.21.15.64:0/753281817 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381029029 0 0) 0x7f15340b0a70 con 0x7f15340b1730 2024-03-25T18:10:12.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.311+0000 7f153159a640 10 monclient(hunting): _negotiate 2024-03-25T18:10:12.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.311+0000 7f153159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:12.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.311+0000 7f153159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:12.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.311+0000 7f153159a640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:10:12.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.311+0000 7f153159a640 1 -- 172.21.15.64:0/753281817 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15340b0d50 con 0x7f15340b1730 2024-03-25T18:10:12.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.312+0000 7f153159a640 1 -- 172.21.15.64:0/753281817 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3957776794 0 0) 0x7f151c003520 con 0x7f153414be90 2024-03-25T18:10:12.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.312+0000 7f153159a640 1 -- 172.21.15.64:0/753281817 >> v1:172.21.15.64:6790/0 conn(0x7f15340b4d60 legacy=0x7f153413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:12.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.312+0000 7f153159a640 1 -- 172.21.15.64:0/753281817 >> v1:172.21.15.86:6789/0 conn(0x7f15340b1730 legacy=0x7f15340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:12.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.312+0000 7f153159a640 10 monclient: _finish_hunting 0 2024-03-25T18:10:12.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.312+0000 7f153159a640 1 monclient: found mon.noname-a 2024-03-25T18:10:12.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.312+0000 7f153159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:12.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.312+0000 7f153159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:10:12.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.312+0000 7f153159a640 1 -- 172.21.15.64:0/753281817 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15340b1030 con 0x7f153414be90 2024-03-25T18:10:12.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.312+0000 7f153159a640 10 monclient: _finish_auth 0 2024-03-25T18:10:12.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.312+0000 7f153159a640 10 monclient: _check_auth_tickets 2024-03-25T18:10:12.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.312+0000 7f153159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:12.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.312+0000 7f153159a640 1 -- 172.21.15.64:0/753281817 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f152c001a10 con 0x7f153414be90 2024-03-25T18:10:12.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.312+0000 7f153159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:12.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.312+0000 7f153159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:12.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.312+0000 7f153159a640 10 monclient: dump: 2024-03-25T18:10:12.316 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:12.317 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:12.317 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:12.317 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:12.317 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:12.317 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:12.317 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:12.317 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:12.317 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:12.317 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:12.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.312+0000 7f1538e57640 5 monclient: authenticate success, global_id 4224 2024-03-25T18:10:12.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.312+0000 7f1538e57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:12.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.315+0000 7f153159a640 1 -- 172.21.15.64:0/753281817 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f152c001c30 con 0x7f153414be90 2024-03-25T18:10:12.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.315+0000 7f153159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:12.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.315+0000 7f153159a640 1 -- 172.21.15.64:0/753281817 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f152c001fc0 con 0x7f153414be90 2024-03-25T18:10:12.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.315+0000 7f153159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:12.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.315+0000 7f153159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:12.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.315+0000 7f153159a640 10 monclient: dump: 2024-03-25T18:10:12.320 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:12.320 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:12.320 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:12.320 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:12.320 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:12.320 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:12.320 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:12.321 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:12.321 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:12.321 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:12.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.315+0000 7f1538e57640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:12.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.315+0000 7f1538e57640 4 set_mon_vals no callback set 2024-03-25T18:10:12.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.316+0000 7f1538e57640 10 monclient: shutdown 2024-03-25T18:10:12.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.316+0000 7f1538e57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:12.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.316+0000 7f1538e57640 1 -- 172.21.15.64:0/753281817 >> v1:172.21.15.64:6789/0 conn(0x7f153414be90 legacy=0x7f153414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:12.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.316+0000 7f1538e57640 1 -- 172.21.15.64:0/753281817 shutdown_connections 2024-03-25T18:10:12.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.316+0000 7f1538e57640 1 -- 172.21.15.64:0/753281817 >> 172.21.15.64:0/753281817 conn(0x7f15340617a0 msgr2=0x7f1534061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:12.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.316+0000 7f1538e57640 1 -- 172.21.15.64:0/753281817 shutdown_connections 2024-03-25T18:10:12.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.316+0000 7f1538e57640 1 -- 172.21.15.64:0/753281817 wait complete. 2024-03-25T18:10:12.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.318+0000 7f1538e57640 10 monclient: build_initial_monmap 2024-03-25T18:10:12.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.318+0000 7f1538e57640 10 monclient: monmap: 2024-03-25T18:10:12.322 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:10:12.322 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:12.322 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:10:12.322 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:10:12.322 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:12.322 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:10:12.322 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:10:12.322 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:10:12.322 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:10:12.322 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:12.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.318+0000 7f1538e57640 1 librados: starting msgr at 2024-03-25T18:10:12.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.318+0000 7f1538e57640 1 librados: starting objecter 2024-03-25T18:10:12.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.318+0000 7f1538e57640 1 Processor -- start 2024-03-25T18:10:12.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.318+0000 7f1538e57640 1 -- start start 2024-03-25T18:10:12.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.318+0000 7f1538e57640 1 librados: setting wanted keys 2024-03-25T18:10:12.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.318+0000 7f1538e57640 1 librados: calling monclient init 2024-03-25T18:10:12.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.318+0000 7f1538e57640 10 monclient: init 2024-03-25T18:10:12.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.319+0000 7f1538e57640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:12.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.319+0000 7f1538e57640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:12.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.319+0000 7f1538e57640 10 monclient(hunting): picked mon.noname-c con 0x7f15340b1730 addr v1:172.21.15.64:6790/0 2024-03-25T18:10:12.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.319+0000 7f1538e57640 10 monclient(hunting): picked mon.noname-b con 0x7f15340b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:10:12.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.319+0000 7f1538e57640 10 monclient(hunting): picked mon.noname-a con 0x7f153414be90 addr v1:172.21.15.64:6789/0 2024-03-25T18:10:12.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.319+0000 7f1538e57640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1534134f90 con 0x7f153414be90 2024-03-25T18:10:12.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.319+0000 7f1538e57640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1534141130 con 0x7f15340b4d60 2024-03-25T18:10:12.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.319+0000 7f1538e57640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1534137fe0 con 0x7f15340b1730 2024-03-25T18:10:12.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.320+0000 7f1538e57640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:12.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.320+0000 7f1538e57640 10 monclient(hunting): authenticate will time out at 6998.559534s 2024-03-25T18:10:12.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.320+0000 7f153259c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f15340b1730 0x7f1534135c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:49688/0 (socket says 172.21.15.64:49688) 2024-03-25T18:10:12.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.320+0000 7f153259c640 1 -- 172.21.15.64:0/3659022722 learned_addr learned my addr 172.21.15.64:0/3659022722 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:10:12.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.320+0000 7f1522ffd640 10 client.?.objecter ms_handle_connect 0x7f153414be90 2024-03-25T18:10:12.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.320+0000 7f1522ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.320+0000 7f1522ffd640 10 client.?.objecter ms_handle_connect 0x7f15340b1730 2024-03-25T18:10:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.320+0000 7f1522ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.320+0000 7f1522ffd640 10 client.?.objecter ms_handle_connect 0x7f15340b4d60 2024-03-25T18:10:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.320+0000 7f1522ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.322+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557469663 0 0) 0x7f1534134f90 con 0x7f153414be90 2024-03-25T18:10:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.323+0000 7f1522ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.323+0000 7f1522ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.323+0000 7f1522ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.323+0000 7f1522ffd640 10 monclient(hunting): my global_id is 4227 2024-03-25T18:10:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.323+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1514004eb0 con 0x7f153414be90 2024-03-25T18:10:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.323+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17905168 0 0) 0x7f1534137fe0 con 0x7f15340b1730 2024-03-25T18:10:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.323+0000 7f1522ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.323+0000 7f1522ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.323+0000 7f1522ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.323+0000 7f1522ffd640 10 monclient(hunting): my global_id is 4226 2024-03-25T18:10:12.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.323+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1534134f90 con 0x7f15340b1730 2024-03-25T18:10:12.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.324+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761593230 0 0) 0x7f1534141130 con 0x7f15340b4d60 2024-03-25T18:10:12.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.324+0000 7f1522ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:12.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.324+0000 7f1522ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:12.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.324+0000 7f1522ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:12.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.324+0000 7f1522ffd640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:10:12.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.324+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1534137fe0 con 0x7f15340b4d60 2024-03-25T18:10:12.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.329+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 213625836 0 0) 0x7f1534134f90 con 0x7f15340b1730 2024-03-25T18:10:12.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.329+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1534141130 con 0x7f15340b1730 2024-03-25T18:10:12.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.330+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4089327015 0 0) 0x7f1514004eb0 con 0x7f153414be90 2024-03-25T18:10:12.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.330+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1514005a30 con 0x7f153414be90 2024-03-25T18:10:12.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.330+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f152c001f40 con 0x7f153414be90 2024-03-25T18:10:12.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.330+0000 7f1522ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:12.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.330+0000 7f1522ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:12.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.330+0000 7f1522ffd640 10 monclient(hunting): dump: 2024-03-25T18:10:12.332 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:12.332 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:12.332 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:12.333 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:12.333 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:12.333 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:12.333 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:12.333 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:12.333 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:12.333 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:12.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.330+0000 7f1522ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:12.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.331+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2699579730 0 0) 0x7f1534137fe0 con 0x7f15340b4d60 2024-03-25T18:10:12.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.331+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1514004eb0 con 0x7f15340b4d60 2024-03-25T18:10:12.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.331+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1528001830 con 0x7f15340b4d60 2024-03-25T18:10:12.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.331+0000 7f1522ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:12.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.331+0000 7f1522ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:12.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.331+0000 7f1522ffd640 10 monclient(hunting): dump: 2024-03-25T18:10:12.334 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:12.334 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:12.334 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:12.334 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:12.335 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:12.335 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:12.335 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:12.335 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:12.335 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:12.335 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:12.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4239352967 0 0) 0x7f1514005a30 con 0x7f153414be90 2024-03-25T18:10:12.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 >> v1:172.21.15.64:6790/0 conn(0x7f15340b1730 legacy=0x7f1534135c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:12.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 >> v1:172.21.15.86:6789/0 conn(0x7f15340b4d60 legacy=0x7f1534134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:12.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1522ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:10:12.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1522ffd640 1 monclient: found mon.a 2024-03-25T18:10:12.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1522ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:12.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1522ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:10:12.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1534138500 con 0x7f153414be90 2024-03-25T18:10:12.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1522ffd640 10 monclient: _finish_auth 0 2024-03-25T18:10:12.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1522ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:10:12.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1522ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:12.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1538e57640 5 monclient: authenticate success, global_id 4227 2024-03-25T18:10:12.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1538e57640 10 monclient: _renew_subs 2024-03-25T18:10:12.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1538e57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:10:12.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1538e57640 1 -- 172.21.15.64:0/3659022722 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1534137c00 con 0x7f153414be90 2024-03-25T18:10:12.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1538e57640 10 client.4227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:12.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1538e57640 10 monclient: _renew_subs 2024-03-25T18:10:12.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1538e57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:10:12.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1538e57640 1 -- 172.21.15.64:0/3659022722 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1534138a00 con 0x7f153414be90 2024-03-25T18:10:12.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.335+0000 7f1538e57640 1 librados: init done 2024-03-25T18:10:12.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.336+0000 7f1538e57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:12.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.336+0000 7f1538e57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:12.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.336+0000 7f1538e57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:10:12.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.336+0000 7f1538e57640 1 -- 172.21.15.64:0/3659022722 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1534138170 con 0x7f153414be90 2024-03-25T18:10:12.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.358+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f152c001420 con 0x7f153414be90 2024-03-25T18:10:12.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.358+0000 7f1522ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:12.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.358+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f152c004fb0 con 0x7f153414be90 2024-03-25T18:10:12.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.358+0000 7f1522ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:12.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.358+0000 7f1522ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:12.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.358+0000 7f1522ffd640 10 monclient: dump: 2024-03-25T18:10:12.360 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:12.360 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:12.360 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:12.360 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:12.360 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:12.360 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:12.360 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:12.360 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:12.360 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:12.360 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:12.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.358+0000 7f1530d99640 4 set_mon_vals no callback set 2024-03-25T18:10:12.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.358+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f152c01af80 con 0x7f153414be90 2024-03-25T18:10:12.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.359+0000 7f153259c640 10 monclient: get_auth_request con 0x7f151406e410 auth_method 0 2024-03-25T18:10:12.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.359+0000 7f1522ffd640 10 client.4227.objecter ms_handle_connect 0x7f151406e410 2024-03-25T18:10:12.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.731+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f152c0549a0 con 0x7f153414be90 2024-03-25T18:10:12.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.731+0000 7f1522ffd640 10 client.4227.objecter ms_dispatch 0x7f1534000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:12.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.731+0000 7f1522ffd640 3 client.4227.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:12.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.731+0000 7f1522ffd640 3 client.4227.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:12.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.731+0000 7f1522ffd640 20 client.4227.objecter dump_active .. 0 homeless 2024-03-25T18:10:12.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.731+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f152c0504b0 con 0x7f153414be90 2024-03-25T18:10:12.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.731+0000 7f1522ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:12.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:12.731+0000 7f1522ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:13.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.002+0000 7f1538e57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:13.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.002+0000 7f1538e57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:13.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.002+0000 7f1538e57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:10:13.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.002+0000 7f1538e57640 1 -- 172.21.15.64:0/3659022722 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15340627b0 con 0x7f153414be90 2024-03-25T18:10:13.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.072+0000 7f1522ffd640 1 -- 172.21.15.64:0/3659022722 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f152c0523e0 con 0x7f153414be90 2024-03-25T18:10:13.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.072+0000 7f1522ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:13.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.072+0000 7f1522ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:13.074 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:10:13.074 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:13.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.074+0000 7f1538e57640 10 librados: watch_flush enter 2024-03-25T18:10:13.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.074+0000 7f1538e57640 10 librados: watch_flush exit 2024-03-25T18:10:13.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.074+0000 7f1538e57640 20 client.4227.objecter shutdown clearing up homeless session... 2024-03-25T18:10:13.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.074+0000 7f1538e57640 10 client.4227.objecter successfully canceled tick 2024-03-25T18:10:13.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.074+0000 7f1538e57640 1 -- 172.21.15.64:0/3659022722 >> v1:172.21.15.64:6800/2915543638 conn(0x7f151406e410 legacy=0x7f15140708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:13.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.074+0000 7f1538e57640 10 monclient: shutdown 2024-03-25T18:10:13.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.074+0000 7f1538e57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:13.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.074+0000 7f1538e57640 1 -- 172.21.15.64:0/3659022722 >> v1:172.21.15.64:6789/0 conn(0x7f153414be90 legacy=0x7f1534134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:13.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.074+0000 7f1538e57640 1 -- 172.21.15.64:0/3659022722 shutdown_connections 2024-03-25T18:10:13.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.074+0000 7f1538e57640 1 -- 172.21.15.64:0/3659022722 >> 172.21.15.64:0/3659022722 conn(0x7f15340617a0 msgr2=0x7f15340b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:13.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.074+0000 7f1538e57640 1 -- 172.21.15.64:0/3659022722 shutdown_connections 2024-03-25T18:10:13.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.074+0000 7f1538e57640 1 -- 172.21.15.64:0/3659022722 wait complete. 2024-03-25T18:10:13.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:13.074+0000 7f1538e57640 1 librados: shutdown 2024-03-25T18:10:13.087 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:14.423 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-03-25T18:10:14.419+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:10:14.938 INFO:tasks.ceph.osd.6.smithi086.stderr:2024-03-25T18:10:14.934+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:10:14.945 INFO:tasks.ceph.osd.7.smithi086.stderr:2024-03-25T18:10:14.942+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:10:15.192 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-03-25T18:10:15.188+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:10:15.732 INFO:tasks.ceph.osd.2.smithi064.stderr:2024-03-25T18:10:15.727+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:10:16.397 INFO:tasks.ceph.osd.1.smithi064.stderr:2024-03-25T18:10:16.392+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:10:16.414 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-03-25T18:10:16.409+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:10:17.242 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-03-25T18:10:17.238+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:10:19.894 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:20.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.011+0000 7ff99de31640 10 monclient: get_monmap_and_config 2024-03-25T18:10:20.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.011+0000 7ff99de31640 10 monclient: build_initial_monmap 2024-03-25T18:10:20.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.011+0000 7ff99de31640 10 monclient: monmap: 2024-03-25T18:10:20.012 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:10:20.012 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:20.012 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:10:20.012130+0000 2024-03-25T18:10:20.012 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:10:20.012130+0000 2024-03-25T18:10:20.012 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:20.012 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:20.012 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:10:20.013 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:10:20.013 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:10:20.013 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:20.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.012+0000 7ff99de31640 1 Processor -- start 2024-03-25T18:10:20.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.012+0000 7ff99de31640 1 -- start start 2024-03-25T18:10:20.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.012+0000 7ff99de31640 10 monclient: init 2024-03-25T18:10:20.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.013+0000 7ff99de31640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:20.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.013+0000 7ff99de31640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:10:20.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.013+0000 7ff99de31640 10 monclient(hunting): picked mon.noname-a con 0x7ff99806ea40 addr v1:172.21.15.64:6789/0 2024-03-25T18:10:20.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.013+0000 7ff99de31640 10 monclient(hunting): picked mon.noname-c con 0x7ff998071800 addr v1:172.21.15.86:6789/0 2024-03-25T18:10:20.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.013+0000 7ff99de31640 10 monclient(hunting): picked mon.noname-b con 0x7ff998071c40 addr v1:172.21.15.64:6790/0 2024-03-25T18:10:20.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.013+0000 7ff99de31640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff99806f1f0 con 0x7ff99806ea40 2024-03-25T18:10:20.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.013+0000 7ff99de31640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff998070d00 con 0x7ff998071800 2024-03-25T18:10:20.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.013+0000 7ff99de31640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9980720b0 con 0x7ff998071c40 2024-03-25T18:10:20.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.013+0000 7ff99de31640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:20.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.013+0000 7ff99de31640 10 monclient(hunting): authenticate will time out at 7006.251989s 2024-03-25T18:10:20.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.026+0000 7ff997fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7ff998071c40 0x7ff99814a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:46790/0 (socket says 172.21.15.64:46790) 2024-03-25T18:10:20.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.026+0000 7ff997fff640 1 -- 172.21.15.64:0/3669006995 learned_addr learned my addr 172.21.15.64:0/3669006995 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:10:20.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.042+0000 7ff9967fc640 1 -- 172.21.15.64:0/3669006995 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2857269566 0 0) 0x7ff9980720b0 con 0x7ff998071c40 2024-03-25T18:10:20.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.043+0000 7ff9967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:20.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.043+0000 7ff9967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:20.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.043+0000 7ff9967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:20.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.043+0000 7ff9967fc640 10 monclient(hunting): my global_id is 4253 2024-03-25T18:10:20.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.043+0000 7ff9967fc640 1 -- 172.21.15.64:0/3669006995 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff980003520 con 0x7ff998071c40 2024-03-25T18:10:20.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.044+0000 7ff9967fc640 1 -- 172.21.15.64:0/3669006995 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240894614 0 0) 0x7ff998070d00 con 0x7ff998071800 2024-03-25T18:10:20.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.044+0000 7ff9967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:20.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.044+0000 7ff9967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:20.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.044+0000 7ff9967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:20.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.044+0000 7ff9967fc640 10 monclient(hunting): my global_id is 4243 2024-03-25T18:10:20.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.044+0000 7ff9967fc640 1 -- 172.21.15.64:0/3669006995 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9980720b0 con 0x7ff998071800 2024-03-25T18:10:20.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.049+0000 7ff9967fc640 1 -- 172.21.15.64:0/3669006995 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1282924624 0 0) 0x7ff980003520 con 0x7ff998071c40 2024-03-25T18:10:20.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.049+0000 7ff9967fc640 1 -- 172.21.15.64:0/3669006995 >> v1:172.21.15.86:6789/0 conn(0x7ff998071800 legacy=0x7ff998147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:20.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.049+0000 7ff9967fc640 1 -- 172.21.15.64:0/3669006995 >> v1:172.21.15.64:6789/0 conn(0x7ff99806ea40 legacy=0x7ff99813ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:20.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.049+0000 7ff9967fc640 10 monclient: _finish_hunting 0 2024-03-25T18:10:20.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.049+0000 7ff9967fc640 1 monclient: found mon.noname-b 2024-03-25T18:10:20.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.049+0000 7ff9967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:20.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.049+0000 7ff9967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:10:20.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.049+0000 7ff9967fc640 1 -- 172.21.15.64:0/3669006995 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff998072780 con 0x7ff998071c40 2024-03-25T18:10:20.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.049+0000 7ff9967fc640 10 monclient: _finish_auth 0 2024-03-25T18:10:20.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.049+0000 7ff9967fc640 10 monclient: _check_auth_tickets 2024-03-25T18:10:20.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.049+0000 7ff9967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:20.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.049+0000 7ff9967fc640 1 -- 172.21.15.64:0/3669006995 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ff990001a10 con 0x7ff998071c40 2024-03-25T18:10:20.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.049+0000 7ff9967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:20.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.049+0000 7ff9967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:20.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.049+0000 7ff9967fc640 10 monclient: dump: 2024-03-25T18:10:20.051 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:20.051 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:20.051 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:20.051 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:20.052 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:20.052 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:20.052 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:20.052 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:20.052 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:20.052 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:20.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.049+0000 7ff99de31640 5 monclient: authenticate success, global_id 4253 2024-03-25T18:10:20.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.049+0000 7ff99de31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:20.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.063+0000 7ff9967fc640 1 -- 172.21.15.64:0/3669006995 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff990001b70 con 0x7ff998071c40 2024-03-25T18:10:20.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.063+0000 7ff9967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:20.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.063+0000 7ff9967fc640 1 -- 172.21.15.64:0/3669006995 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ff990001f80 con 0x7ff998071c40 2024-03-25T18:10:20.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.063+0000 7ff99de31640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:20.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.063+0000 7ff99de31640 4 set_mon_vals no callback set 2024-03-25T18:10:20.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.063+0000 7ff9967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:20.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.063+0000 7ff9967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:20.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.063+0000 7ff9967fc640 10 monclient: dump: 2024-03-25T18:10:20.064 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:20.065 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:20.065 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:20.065 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:20.065 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:20.065 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:20.065 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:20.065 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:20.065 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:20.065 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:20.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.064+0000 7ff99de31640 10 monclient: shutdown 2024-03-25T18:10:20.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.064+0000 7ff99de31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:20.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.064+0000 7ff99de31640 1 -- 172.21.15.64:0/3669006995 >> v1:172.21.15.64:6790/0 conn(0x7ff998071c40 legacy=0x7ff99814a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:20.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.064+0000 7ff99de31640 1 -- 172.21.15.64:0/3669006995 shutdown_connections 2024-03-25T18:10:20.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.064+0000 7ff99de31640 1 -- 172.21.15.64:0/3669006995 >> 172.21.15.64:0/3669006995 conn(0x7ff99806b630 msgr2=0x7ff9980b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:20.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.064+0000 7ff99de31640 1 -- 172.21.15.64:0/3669006995 shutdown_connections 2024-03-25T18:10:20.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.064+0000 7ff99de31640 1 -- 172.21.15.64:0/3669006995 wait complete. 2024-03-25T18:10:20.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.065+0000 7ff99de31640 10 monclient: build_initial_monmap 2024-03-25T18:10:20.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.065+0000 7ff99de31640 10 monclient: monmap: 2024-03-25T18:10:20.068 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:10:20.068 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:20.068 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:10:20.068 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:10:20.068 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:20.068 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:10:20.068 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:10:20.068 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:10:20.068 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:10:20.069 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:20.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.065+0000 7ff99de31640 1 librados: starting msgr at 2024-03-25T18:10:20.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.065+0000 7ff99de31640 1 librados: starting objecter 2024-03-25T18:10:20.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.065+0000 7ff99de31640 1 Processor -- start 2024-03-25T18:10:20.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.065+0000 7ff99de31640 1 -- start start 2024-03-25T18:10:20.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.065+0000 7ff99de31640 1 librados: setting wanted keys 2024-03-25T18:10:20.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.065+0000 7ff99de31640 1 librados: calling monclient init 2024-03-25T18:10:20.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.065+0000 7ff99de31640 10 monclient: init 2024-03-25T18:10:20.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.066+0000 7ff99de31640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:20.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.066+0000 7ff99de31640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:10:20.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.066+0000 7ff99de31640 10 monclient(hunting): picked mon.noname-c con 0x7ff99806ea40 addr v1:172.21.15.64:6790/0 2024-03-25T18:10:20.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.066+0000 7ff99de31640 10 monclient(hunting): picked mon.noname-a con 0x7ff998071800 addr v1:172.21.15.64:6789/0 2024-03-25T18:10:20.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.066+0000 7ff99de31640 10 monclient(hunting): picked mon.noname-b con 0x7ff998071c40 addr v1:172.21.15.86:6789/0 2024-03-25T18:10:20.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.066+0000 7ff99de31640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff99811d560 con 0x7ff998071800 2024-03-25T18:10:20.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.066+0000 7ff99de31640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff99813da20 con 0x7ff998071c40 2024-03-25T18:10:20.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.066+0000 7ff99de31640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9981d5b00 con 0x7ff99806ea40 2024-03-25T18:10:20.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.066+0000 7ff99de31640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:20.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.066+0000 7ff99de31640 10 monclient(hunting): authenticate will time out at 7006.305232s 2024-03-25T18:10:20.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.071+0000 7ff997fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff998071c40 0x7ff99813d310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:53224/0 (socket says 172.21.15.64:53224) 2024-03-25T18:10:20.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.071+0000 7ff997fff640 1 -- 172.21.15.64:0/2751242494 learned_addr learned my addr 172.21.15.64:0/2751242494 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:10:20.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.076+0000 7ff97ffff640 10 client.?.objecter ms_handle_connect 0x7ff998071c40 2024-03-25T18:10:20.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.076+0000 7ff97ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:20.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.081+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740949119 0 0) 0x7ff99813da20 con 0x7ff998071c40 2024-03-25T18:10:20.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.081+0000 7ff97ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:20.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.081+0000 7ff97ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:20.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.081+0000 7ff97ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:20.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.081+0000 7ff97ffff640 10 monclient(hunting): my global_id is 4246 2024-03-25T18:10:20.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.081+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff984004eb0 con 0x7ff998071c40 2024-03-25T18:10:20.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.082+0000 7ff97ffff640 10 client.?.objecter ms_handle_connect 0x7ff99806ea40 2024-03-25T18:10:20.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.082+0000 7ff97ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:20.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.085+0000 7ff97ffff640 10 client.?.objecter ms_handle_connect 0x7ff998071800 2024-03-25T18:10:20.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.085+0000 7ff97ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:20.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.091+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926523958 0 0) 0x7ff9981d5b00 con 0x7ff99806ea40 2024-03-25T18:10:20.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.091+0000 7ff97ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:20.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.091+0000 7ff97ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:20.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.091+0000 7ff97ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:20.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.091+0000 7ff97ffff640 10 monclient(hunting): my global_id is 4256 2024-03-25T18:10:20.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.091+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff99813da20 con 0x7ff99806ea40 2024-03-25T18:10:20.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.095+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895554437 0 0) 0x7ff99811d560 con 0x7ff998071800 2024-03-25T18:10:20.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.095+0000 7ff97ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:20.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.095+0000 7ff97ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:20.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.095+0000 7ff97ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:20.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.095+0000 7ff97ffff640 10 monclient(hunting): my global_id is 4257 2024-03-25T18:10:20.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.095+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9981d5b00 con 0x7ff998071800 2024-03-25T18:10:20.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.103+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 290573788 0 0) 0x7ff99813da20 con 0x7ff99806ea40 2024-03-25T18:10:20.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.103+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff99811d560 con 0x7ff99806ea40 2024-03-25T18:10:20.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.103+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ff9880018a0 con 0x7ff99806ea40 2024-03-25T18:10:20.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.103+0000 7ff97ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:20.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.103+0000 7ff97ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:20.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.103+0000 7ff97ffff640 10 monclient(hunting): dump: 2024-03-25T18:10:20.104 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:20.104 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:20.104 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:20.104 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:20.104 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:20.104 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:20.105 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:20.105 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:20.105 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:20.105 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:20.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.103+0000 7ff97ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:20.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.104+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2477820631 0 0) 0x7ff984004eb0 con 0x7ff998071c40 2024-03-25T18:10:20.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.104+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff99813da20 con 0x7ff998071c40 2024-03-25T18:10:20.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.104+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ff990001f40 con 0x7ff998071c40 2024-03-25T18:10:20.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.104+0000 7ff97ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:20.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.104+0000 7ff97ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:20.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.104+0000 7ff97ffff640 10 monclient(hunting): dump: 2024-03-25T18:10:20.106 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:20.106 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:20.106 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:20.106 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:20.106 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:20.106 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:20.106 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:20.106 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:20.106 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:20.107 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:20.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2883793456 0 0) 0x7ff99813da20 con 0x7ff998071c40 2024-03-25T18:10:20.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 >> v1:172.21.15.64:6790/0 conn(0x7ff99806ea40 legacy=0x7ff9981d53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:20.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 >> v1:172.21.15.64:6789/0 conn(0x7ff998071800 legacy=0x7ff99813cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:20.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff97ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:10:20.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff97ffff640 1 monclient: found mon.b 2024-03-25T18:10:20.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff97ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:20.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff97ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:20.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99811c100 con 0x7ff998071c40 2024-03-25T18:10:20.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff97ffff640 10 monclient: _finish_auth 0 2024-03-25T18:10:20.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff97ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:10:20.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff97ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:20.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff99de31640 5 monclient: authenticate success, global_id 4246 2024-03-25T18:10:20.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff99de31640 10 monclient: _renew_subs 2024-03-25T18:10:20.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff99de31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:20.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff99de31640 1 -- 172.21.15.64:0/2751242494 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff99811c350 con 0x7ff998071c40 2024-03-25T18:10:20.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff99de31640 10 client.4246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:20.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff99de31640 10 monclient: _renew_subs 2024-03-25T18:10:20.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff99de31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:20.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff99de31640 1 -- 172.21.15.64:0/2751242494 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9980ad350 con 0x7ff998071c40 2024-03-25T18:10:20.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.115+0000 7ff99de31640 1 librados: init done 2024-03-25T18:10:20.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.116+0000 7ff99de31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:20.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.116+0000 7ff99de31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:20.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.116+0000 7ff99de31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:20.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.116+0000 7ff99de31640 1 -- 172.21.15.64:0/2751242494 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9980627b0 con 0x7ff998071c40 2024-03-25T18:10:20.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.393+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff990004e10 con 0x7ff998071c40 2024-03-25T18:10:20.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.393+0000 7ff97ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:20.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.393+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ff990005120 con 0x7ff998071c40 2024-03-25T18:10:20.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.393+0000 7ff97ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:20.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.393+0000 7ff97ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:20.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.393+0000 7ff97ffff640 10 monclient: dump: 2024-03-25T18:10:20.394 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:20.394 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:20.394 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:20.394 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:20.395 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:20.395 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:20.395 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:20.395 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:20.395 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:20.395 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:20.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.393+0000 7ff995ffb640 4 set_mon_vals no callback set 2024-03-25T18:10:20.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.394+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7ff99001b0f0 con 0x7ff998071c40 2024-03-25T18:10:20.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.395+0000 7ff9977fe640 10 monclient: get_auth_request con 0x7ff98406df70 auth_method 0 2024-03-25T18:10:20.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.395+0000 7ff97ffff640 10 client.4246.objecter ms_handle_connect 0x7ff98406df70 2024-03-25T18:10:20.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.399+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7ff990054c00 con 0x7ff998071c40 2024-03-25T18:10:20.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.399+0000 7ff97ffff640 10 client.4246.objecter ms_dispatch 0x7ff998000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:20.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.399+0000 7ff97ffff640 3 client.4246.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:20.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.399+0000 7ff97ffff640 3 client.4246.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:20.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.399+0000 7ff97ffff640 20 client.4246.objecter dump_active .. 0 homeless 2024-03-25T18:10:20.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.399+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff990050710 con 0x7ff998071c40 2024-03-25T18:10:20.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.399+0000 7ff97ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:20.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.399+0000 7ff97ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:20.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.649+0000 7ff99de31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:20.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.649+0000 7ff99de31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:20.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.649+0000 7ff99de31640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:20.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.649+0000 7ff99de31640 1 -- 172.21.15.64:0/2751242494 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9980681e0 con 0x7ff998071c40 2024-03-25T18:10:20.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.677+0000 7ff97ffff640 1 -- 172.21.15.64:0/2751242494 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7ff990052640 con 0x7ff998071c40 2024-03-25T18:10:20.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.677+0000 7ff97ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:20.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.677+0000 7ff97ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:20.678 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:10:20.678 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:20.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.679+0000 7ff99de31640 10 librados: watch_flush enter 2024-03-25T18:10:20.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.679+0000 7ff99de31640 10 librados: watch_flush exit 2024-03-25T18:10:20.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.679+0000 7ff99de31640 20 client.4246.objecter shutdown clearing up homeless session... 2024-03-25T18:10:20.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.679+0000 7ff99de31640 10 client.4246.objecter successfully canceled tick 2024-03-25T18:10:20.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.679+0000 7ff99de31640 1 -- 172.21.15.64:0/2751242494 >> v1:172.21.15.64:6800/2915543638 conn(0x7ff98406df70 legacy=0x7ff984070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:20.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.680+0000 7ff99de31640 10 monclient: shutdown 2024-03-25T18:10:20.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.680+0000 7ff99de31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:20.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.680+0000 7ff99de31640 1 -- 172.21.15.64:0/2751242494 >> v1:172.21.15.86:6789/0 conn(0x7ff998071c40 legacy=0x7ff99813d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:20.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.680+0000 7ff99de31640 1 -- 172.21.15.64:0/2751242494 shutdown_connections 2024-03-25T18:10:20.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.680+0000 7ff99de31640 1 -- 172.21.15.64:0/2751242494 >> 172.21.15.64:0/2751242494 conn(0x7ff99806b630 msgr2=0x7ff9980b10a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:20.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.680+0000 7ff99de31640 1 -- 172.21.15.64:0/2751242494 shutdown_connections 2024-03-25T18:10:20.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.680+0000 7ff99de31640 1 -- 172.21.15.64:0/2751242494 wait complete. 2024-03-25T18:10:20.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:20.680+0000 7ff99de31640 1 librados: shutdown 2024-03-25T18:10:20.693 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:27.499 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:27.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.618+0000 7fd6b98f4640 10 monclient: get_monmap_and_config 2024-03-25T18:10:27.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.618+0000 7fd6b98f4640 10 monclient: build_initial_monmap 2024-03-25T18:10:27.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.619+0000 7fd6b98f4640 10 monclient: monmap: 2024-03-25T18:10:27.620 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:10:27.620 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:27.620 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:10:27.619876+0000 2024-03-25T18:10:27.620 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:10:27.619876+0000 2024-03-25T18:10:27.620 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:27.620 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:27.620 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:10:27.621 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:10:27.621 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:10:27.621 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:27.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.619+0000 7fd6b98f4640 1 Processor -- start 2024-03-25T18:10:27.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.619+0000 7fd6b98f4640 1 -- start start 2024-03-25T18:10:27.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.619+0000 7fd6b98f4640 10 monclient: init 2024-03-25T18:10:27.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.620+0000 7fd6b98f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:27.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.620+0000 7fd6b98f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:10:27.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.620+0000 7fd6b98f4640 10 monclient(hunting): picked mon.noname-c con 0x7fd6b40a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:10:27.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.620+0000 7fd6b98f4640 10 monclient(hunting): picked mon.noname-b con 0x7fd6b413f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:10:27.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.620+0000 7fd6b98f4640 10 monclient(hunting): picked mon.noname-a con 0x7fd6b41430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:10:27.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.620+0000 7fd6b98f4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b4142520 con 0x7fd6b41430c0 2024-03-25T18:10:27.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.620+0000 7fd6b98f4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b40b28a0 con 0x7fd6b40a98f0 2024-03-25T18:10:27.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.620+0000 7fd6b98f4640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b40b2a30 con 0x7fd6b413f710 2024-03-25T18:10:27.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.620+0000 7fd6b98f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:27.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.620+0000 7fd6b98f4640 10 monclient(hunting): authenticate will time out at 7013.859155s 2024-03-25T18:10:27.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.633+0000 7fd6b2ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd6b40a98f0 0x7fd6b40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:36034/0 (socket says 172.21.15.64:36034) 2024-03-25T18:10:27.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.633+0000 7fd6b2ffd640 1 -- 172.21.15.64:0/47879611 learned_addr learned my addr 172.21.15.64:0/47879611 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:10:27.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.642+0000 7fd6b1ffb640 1 -- 172.21.15.64:0/47879611 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127464884 0 0) 0x7fd6b40b28a0 con 0x7fd6b40a98f0 2024-03-25T18:10:27.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.642+0000 7fd6b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:27.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.642+0000 7fd6b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:27.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.642+0000 7fd6b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:27.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.642+0000 7fd6b1ffb640 10 monclient(hunting): my global_id is 4249 2024-03-25T18:10:27.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.642+0000 7fd6b1ffb640 1 -- 172.21.15.64:0/47879611 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd69c003520 con 0x7fd6b40a98f0 2024-03-25T18:10:27.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.650+0000 7fd6b1ffb640 1 -- 172.21.15.64:0/47879611 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070865859 0 0) 0x7fd6b40b2a30 con 0x7fd6b413f710 2024-03-25T18:10:27.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.650+0000 7fd6b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:27.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.650+0000 7fd6b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:27.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.650+0000 7fd6b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:27.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.650+0000 7fd6b1ffb640 10 monclient(hunting): my global_id is 4259 2024-03-25T18:10:27.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.650+0000 7fd6b1ffb640 1 -- 172.21.15.64:0/47879611 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6b40b28a0 con 0x7fd6b413f710 2024-03-25T18:10:27.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.650+0000 7fd6b1ffb640 1 -- 172.21.15.64:0/47879611 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109422934 0 0) 0x7fd6b4142520 con 0x7fd6b41430c0 2024-03-25T18:10:27.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.650+0000 7fd6b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:27.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.650+0000 7fd6b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:27.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.650+0000 7fd6b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:27.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.650+0000 7fd6b1ffb640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:10:27.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.650+0000 7fd6b1ffb640 1 -- 172.21.15.64:0/47879611 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6b40b2a30 con 0x7fd6b41430c0 2024-03-25T18:10:27.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.666+0000 7fd6b1ffb640 1 -- 172.21.15.64:0/47879611 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 283248420 0 0) 0x7fd69c003520 con 0x7fd6b40a98f0 2024-03-25T18:10:27.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.666+0000 7fd6b1ffb640 1 -- 172.21.15.64:0/47879611 >> v1:172.21.15.64:6790/0 conn(0x7fd6b413f710 legacy=0x7fd6b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:27.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.666+0000 7fd6b1ffb640 1 -- 172.21.15.64:0/47879611 >> v1:172.21.15.64:6789/0 conn(0x7fd6b41430c0 legacy=0x7fd6b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:27.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.666+0000 7fd6b1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:10:27.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.666+0000 7fd6b1ffb640 1 monclient: found mon.noname-c 2024-03-25T18:10:27.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.666+0000 7fd6b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:27.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.666+0000 7fd6b1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:10:27.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.666+0000 7fd6b1ffb640 1 -- 172.21.15.64:0/47879611 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b40b44c0 con 0x7fd6b40a98f0 2024-03-25T18:10:27.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.666+0000 7fd6b1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:10:27.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.666+0000 7fd6b1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:10:27.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.666+0000 7fd6b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:27.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.666+0000 7fd6b1ffb640 1 -- 172.21.15.64:0/47879611 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fd6a4001690 con 0x7fd6b40a98f0 2024-03-25T18:10:27.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.666+0000 7fd6b98f4640 5 monclient: authenticate success, global_id 4249 2024-03-25T18:10:27.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.666+0000 7fd6b98f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:27.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.666+0000 7fd6b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:27.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.666+0000 7fd6b1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:27.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.666+0000 7fd6b1ffb640 10 monclient: dump: 2024-03-25T18:10:27.668 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:27.668 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:27.668 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:27.668 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:27.668 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:27.668 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:27.668 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:27.668 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:27.668 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:27.668 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:27.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.666+0000 7fd6b98f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:27.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.678+0000 7fd6b1ffb640 1 -- 172.21.15.64:0/47879611 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6a40017f0 con 0x7fd6b40a98f0 2024-03-25T18:10:27.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.678+0000 7fd6b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:27.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.678+0000 7fd6b98f4640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:27.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.678+0000 7fd6b98f4640 4 set_mon_vals no callback set 2024-03-25T18:10:27.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.678+0000 7fd6b98f4640 10 monclient: shutdown 2024-03-25T18:10:27.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.678+0000 7fd6b98f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:27.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.678+0000 7fd6b98f4640 1 -- 172.21.15.64:0/47879611 >> v1:172.21.15.86:6789/0 conn(0x7fd6b40a98f0 legacy=0x7fd6b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:27.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.679+0000 7fd6b98f4640 1 -- 172.21.15.64:0/47879611 shutdown_connections 2024-03-25T18:10:27.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.679+0000 7fd6b98f4640 1 -- 172.21.15.64:0/47879611 >> 172.21.15.64:0/47879611 conn(0x7fd6b40b7ea0 msgr2=0x7fd6b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:27.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.679+0000 7fd6b98f4640 1 -- 172.21.15.64:0/47879611 shutdown_connections 2024-03-25T18:10:27.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.679+0000 7fd6b98f4640 1 -- 172.21.15.64:0/47879611 wait complete. 2024-03-25T18:10:27.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.680+0000 7fd6b98f4640 10 monclient: build_initial_monmap 2024-03-25T18:10:27.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.680+0000 7fd6b98f4640 10 monclient: monmap: 2024-03-25T18:10:27.681 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:10:27.681 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:27.681 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:10:27.681 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:10:27.681 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:27.681 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:10:27.682 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:10:27.682 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:10:27.682 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:10:27.682 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:27.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.680+0000 7fd6b98f4640 1 librados: starting msgr at 2024-03-25T18:10:27.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.680+0000 7fd6b98f4640 1 librados: starting objecter 2024-03-25T18:10:27.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.680+0000 7fd6b98f4640 1 Processor -- start 2024-03-25T18:10:27.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.680+0000 7fd6b98f4640 1 -- start start 2024-03-25T18:10:27.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.680+0000 7fd6b98f4640 1 librados: setting wanted keys 2024-03-25T18:10:27.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.680+0000 7fd6b98f4640 1 librados: calling monclient init 2024-03-25T18:10:27.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.680+0000 7fd6b98f4640 10 monclient: init 2024-03-25T18:10:27.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.681+0000 7fd6b98f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:27.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.681+0000 7fd6b98f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:27.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.681+0000 7fd6b98f4640 10 monclient(hunting): picked mon.noname-c con 0x7fd6b40a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:10:27.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.681+0000 7fd6b98f4640 10 monclient(hunting): picked mon.noname-b con 0x7fd6b413f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:10:27.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.681+0000 7fd6b98f4640 10 monclient(hunting): picked mon.noname-a con 0x7fd6b41430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:10:27.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.681+0000 7fd6b98f4640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b40ab9e0 con 0x7fd6b41430c0 2024-03-25T18:10:27.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.681+0000 7fd6b98f4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b4138920 con 0x7fd6b413f710 2024-03-25T18:10:27.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.681+0000 7fd6b98f4640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b4138ab0 con 0x7fd6b40a98f0 2024-03-25T18:10:27.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.681+0000 7fd6b98f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:27.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.681+0000 7fd6b98f4640 10 monclient(hunting): authenticate will time out at 7013.919963s 2024-03-25T18:10:27.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.690+0000 7fd6b27fc640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd6b413f710 0x7fd6b41dacb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:36084/0 (socket says 172.21.15.64:36084) 2024-03-25T18:10:27.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.690+0000 7fd6b27fc640 1 -- 172.21.15.64:0/301984607 learned_addr learned my addr 172.21.15.64:0/301984607 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:10:27.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.690+0000 7fd69bfff640 10 client.?.objecter ms_handle_connect 0x7fd6b413f710 2024-03-25T18:10:27.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.690+0000 7fd69bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:27.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.720+0000 7fd69bfff640 10 client.?.objecter ms_handle_connect 0x7fd6b41430c0 2024-03-25T18:10:27.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.720+0000 7fd69bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:27.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.721+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1475751249 0 0) 0x7fd6b4138920 con 0x7fd6b413f710 2024-03-25T18:10:27.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.721+0000 7fd69bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:27.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.721+0000 7fd69bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:27.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.721+0000 7fd69bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:27.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.721+0000 7fd69bfff640 10 monclient(hunting): my global_id is 4252 2024-03-25T18:10:27.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.721+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6a0004eb0 con 0x7fd6b413f710 2024-03-25T18:10:27.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.739+0000 7fd69bfff640 10 client.?.objecter ms_handle_connect 0x7fd6b40a98f0 2024-03-25T18:10:27.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.739+0000 7fd69bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:27.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.740+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518995966 0 0) 0x7fd6b40ab9e0 con 0x7fd6b41430c0 2024-03-25T18:10:27.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.740+0000 7fd69bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:27.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.740+0000 7fd69bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:27.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.740+0000 7fd69bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:27.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.740+0000 7fd69bfff640 10 monclient(hunting): my global_id is 4263 2024-03-25T18:10:27.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.740+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6b4138920 con 0x7fd6b41430c0 2024-03-25T18:10:27.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.748+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1887919213 0 0) 0x7fd6a0004eb0 con 0x7fd6b413f710 2024-03-25T18:10:27.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.748+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6b40ab9e0 con 0x7fd6b413f710 2024-03-25T18:10:27.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.748+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fd6a8001a30 con 0x7fd6b413f710 2024-03-25T18:10:27.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.748+0000 7fd69bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:27.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.748+0000 7fd69bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:27.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.748+0000 7fd69bfff640 10 monclient(hunting): dump: 2024-03-25T18:10:27.749 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:27.749 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:27.749 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:27.749 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:27.749 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:27.749 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:27.750 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:27.750 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:27.750 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:27.750 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:27.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.748+0000 7fd69bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:27.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.757+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2178576246 0 0) 0x7fd6b4138920 con 0x7fd6b41430c0 2024-03-25T18:10:27.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.757+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6a0004eb0 con 0x7fd6b41430c0 2024-03-25T18:10:27.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.757+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fd6ac001b50 con 0x7fd6b41430c0 2024-03-25T18:10:27.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.757+0000 7fd69bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:27.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.757+0000 7fd69bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:27.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.757+0000 7fd69bfff640 10 monclient(hunting): dump: 2024-03-25T18:10:27.759 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:27.759 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:27.759 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:27.759 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:27.759 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:27.759 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:27.759 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:27.759 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:27.759 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:27.759 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:27.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2638252988 0 0) 0x7fd6b40ab9e0 con 0x7fd6b413f710 2024-03-25T18:10:27.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 >> v1:172.21.15.64:6790/0 conn(0x7fd6b40a98f0 legacy=0x7fd6b4138210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:27.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 >> v1:172.21.15.64:6789/0 conn(0x7fd6b41430c0 legacy=0x7fd6b41e1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:27.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd69bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:10:27.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd69bfff640 1 monclient: found mon.b 2024-03-25T18:10:27.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd69bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:27.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd69bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:27.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b4137670 con 0x7fd6b413f710 2024-03-25T18:10:27.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd69bfff640 10 monclient: _finish_auth 0 2024-03-25T18:10:27.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd69bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:10:27.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd69bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:27.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd6b98f4640 5 monclient: authenticate success, global_id 4252 2024-03-25T18:10:27.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd6b98f4640 10 monclient: _renew_subs 2024-03-25T18:10:27.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd6b98f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:27.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd6b98f4640 1 -- 172.21.15.64:0/301984607 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b41db3c0 con 0x7fd6b413f710 2024-03-25T18:10:27.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd6b98f4640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:27.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd6b98f4640 10 monclient: _renew_subs 2024-03-25T18:10:27.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd6b98f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:27.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd6b98f4640 1 -- 172.21.15.64:0/301984607 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b41e34d0 con 0x7fd6b413f710 2024-03-25T18:10:27.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.760+0000 7fd6b98f4640 1 librados: init done 2024-03-25T18:10:27.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.762+0000 7fd6b98f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:27.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.762+0000 7fd6b98f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:27.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.762+0000 7fd6b98f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:27.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:27.762+0000 7fd6b98f4640 1 -- 172.21.15.64:0/301984607 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b41378c0 con 0x7fd6b413f710 2024-03-25T18:10:28.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.033+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6a8001c70 con 0x7fd6b413f710 2024-03-25T18:10:28.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.033+0000 7fd69bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:28.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.033+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fd6a8003a60 con 0x7fd6b413f710 2024-03-25T18:10:28.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.033+0000 7fd69bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:28.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.033+0000 7fd69bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:28.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.033+0000 7fd69bfff640 10 monclient: dump: 2024-03-25T18:10:28.034 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:28.034 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:28.034 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:28.034 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:28.034 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:28.034 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:28.035 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:28.035 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:28.035 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:28.035 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:28.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.033+0000 7fd6b17fa640 4 set_mon_vals no callback set 2024-03-25T18:10:28.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.034+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fd6a8003c40 con 0x7fd6b413f710 2024-03-25T18:10:28.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.034+0000 7fd6b2ffd640 10 monclient: get_auth_request con 0x7fd6a006de90 auth_method 0 2024-03-25T18:10:28.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.035+0000 7fd69bfff640 10 client.4252.objecter ms_handle_connect 0x7fd6a006de90 2024-03-25T18:10:28.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.039+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fd6a8054900 con 0x7fd6b413f710 2024-03-25T18:10:28.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.039+0000 7fd69bfff640 10 client.4252.objecter ms_dispatch 0x7fd6b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:28.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.039+0000 7fd69bfff640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:28.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.039+0000 7fd69bfff640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:28.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.039+0000 7fd69bfff640 20 client.4252.objecter dump_active .. 0 homeless 2024-03-25T18:10:28.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.039+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd6a8050410 con 0x7fd6b413f710 2024-03-25T18:10:28.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.039+0000 7fd69bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:28.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.039+0000 7fd69bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:28.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.263+0000 7fd6b98f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:28.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.263+0000 7fd6b98f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:28.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.263+0000 7fd6b98f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:28.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.263+0000 7fd6b98f4640 1 -- 172.21.15.64:0/301984607 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6b41db760 con 0x7fd6b413f710 2024-03-25T18:10:28.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.291+0000 7fd69bfff640 1 -- 172.21.15.64:0/301984607 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fd6a8052340 con 0x7fd6b413f710 2024-03-25T18:10:28.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.291+0000 7fd69bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:28.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.291+0000 7fd69bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:28.292 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:10:28.292 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:28.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.293+0000 7fd6b98f4640 10 librados: watch_flush enter 2024-03-25T18:10:28.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.293+0000 7fd6b98f4640 10 librados: watch_flush exit 2024-03-25T18:10:28.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.293+0000 7fd6b98f4640 20 client.4252.objecter shutdown clearing up homeless session... 2024-03-25T18:10:28.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.293+0000 7fd6b98f4640 10 client.4252.objecter successfully canceled tick 2024-03-25T18:10:28.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.293+0000 7fd6b98f4640 1 -- 172.21.15.64:0/301984607 >> v1:172.21.15.64:6800/2915543638 conn(0x7fd6a006de90 legacy=0x7fd6a0070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:28.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.293+0000 7fd6b98f4640 10 monclient: shutdown 2024-03-25T18:10:28.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.293+0000 7fd6b98f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:28.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.293+0000 7fd6b98f4640 1 -- 172.21.15.64:0/301984607 >> v1:172.21.15.86:6789/0 conn(0x7fd6b413f710 legacy=0x7fd6b41dacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:28.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.293+0000 7fd6b98f4640 1 -- 172.21.15.64:0/301984607 shutdown_connections 2024-03-25T18:10:28.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.293+0000 7fd6b98f4640 1 -- 172.21.15.64:0/301984607 >> 172.21.15.64:0/301984607 conn(0x7fd6b40b7ea0 msgr2=0x7fd6b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:28.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.294+0000 7fd6b98f4640 1 -- 172.21.15.64:0/301984607 shutdown_connections 2024-03-25T18:10:28.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.294+0000 7fd6b98f4640 1 -- 172.21.15.64:0/301984607 wait complete. 2024-03-25T18:10:28.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:28.294+0000 7fd6b98f4640 1 librados: shutdown 2024-03-25T18:10:28.307 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:31.373 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:01:14.486 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:31.434 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:01:14.547 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:31.526 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:01:14.639 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:31.569 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:01:14.648 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:31.585 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:01:14.698 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:31.640 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:01:14.719 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:31.680 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:01:14.793 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:31.857 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:01:14.970 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:31.876 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:01:14.955 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:35.114 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:35.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.228+0000 7effebd23640 10 monclient: get_monmap_and_config 2024-03-25T18:10:35.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.228+0000 7effebd23640 10 monclient: build_initial_monmap 2024-03-25T18:10:35.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.228+0000 7effebd23640 10 monclient: monmap: 2024-03-25T18:10:35.229 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:10:35.229 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:35.229 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:10:35.229160+0000 2024-03-25T18:10:35.229 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:10:35.229160+0000 2024-03-25T18:10:35.229 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:35.229 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:35.229 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:10:35.229 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:10:35.229 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:10:35.229 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:35.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.228+0000 7effebd23640 1 Processor -- start 2024-03-25T18:10:35.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.229+0000 7effebd23640 1 -- start start 2024-03-25T18:10:35.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.229+0000 7effebd23640 10 monclient: init 2024-03-25T18:10:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.229+0000 7effebd23640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.230+0000 7effebd23640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.230+0000 7effebd23640 10 monclient(hunting): picked mon.noname-b con 0x7effe40a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:10:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.230+0000 7effebd23640 10 monclient(hunting): picked mon.noname-c con 0x7effe413f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:10:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.230+0000 7effebd23640 10 monclient(hunting): picked mon.noname-a con 0x7effe41430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:10:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.230+0000 7effebd23640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe4142520 con 0x7effe41430c0 2024-03-25T18:10:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.230+0000 7effebd23640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe40b28a0 con 0x7effe413f710 2024-03-25T18:10:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.230+0000 7effebd23640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe40b2a30 con 0x7effe40a98f0 2024-03-25T18:10:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.230+0000 7effebd23640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:35.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.230+0000 7effebd23640 10 monclient(hunting): authenticate will time out at 7021.468579s 2024-03-25T18:10:35.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.236+0000 7effe92be640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7effe413f710 0x7effe41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:40322/0 (socket says 172.21.15.64:40322) 2024-03-25T18:10:35.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.236+0000 7effe92be640 1 -- 172.21.15.64:0/3297993521 learned_addr learned my addr 172.21.15.64:0/3297993521 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:10:35.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.246+0000 7effe8abd640 1 -- 172.21.15.64:0/3297993521 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684250687 0 0) 0x7effe40b28a0 con 0x7effe413f710 2024-03-25T18:10:35.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.246+0000 7effe8abd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:35.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.246+0000 7effe8abd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:35.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.246+0000 7effe8abd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:35.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.246+0000 7effe8abd640 10 monclient(hunting): my global_id is 4255 2024-03-25T18:10:35.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.246+0000 7effe8abd640 1 -- 172.21.15.64:0/3297993521 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effd0003520 con 0x7effe413f710 2024-03-25T18:10:35.271 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.270+0000 7effe8abd640 1 -- 172.21.15.64:0/3297993521 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3635418694 0 0) 0x7effd0003520 con 0x7effe413f710 2024-03-25T18:10:35.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.270+0000 7effe8abd640 1 -- 172.21.15.64:0/3297993521 >> v1:172.21.15.64:6790/0 conn(0x7effe40a98f0 legacy=0x7effe40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:35.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.270+0000 7effe8abd640 1 -- 172.21.15.64:0/3297993521 >> v1:172.21.15.64:6789/0 conn(0x7effe41430c0 legacy=0x7effe414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:35.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.270+0000 7effe8abd640 10 monclient: _finish_hunting 0 2024-03-25T18:10:35.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.270+0000 7effe8abd640 1 monclient: found mon.noname-c 2024-03-25T18:10:35.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.270+0000 7effe8abd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:35.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.270+0000 7effe8abd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:10:35.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.270+0000 7effe8abd640 1 -- 172.21.15.64:0/3297993521 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe40b44c0 con 0x7effe413f710 2024-03-25T18:10:35.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.270+0000 7effe8abd640 10 monclient: _finish_auth 0 2024-03-25T18:10:35.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.270+0000 7effe8abd640 10 monclient: _check_auth_tickets 2024-03-25T18:10:35.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.270+0000 7effe8abd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:35.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.270+0000 7effe8abd640 1 -- 172.21.15.64:0/3297993521 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7effd4001690 con 0x7effe413f710 2024-03-25T18:10:35.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.270+0000 7effebd23640 5 monclient: authenticate success, global_id 4255 2024-03-25T18:10:35.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.270+0000 7effebd23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:35.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.270+0000 7effe8abd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:35.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.270+0000 7effe8abd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:35.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.270+0000 7effe8abd640 10 monclient: dump: 2024-03-25T18:10:35.273 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:35.273 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:35.273 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:35.273 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:35.273 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:35.273 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:35.273 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:35.273 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:35.273 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:35.273 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:35.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.270+0000 7effebd23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:35.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.299+0000 7effe8abd640 1 -- 172.21.15.64:0/3297993521 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effd4001870 con 0x7effe413f710 2024-03-25T18:10:35.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.299+0000 7effe8abd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:35.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.299+0000 7effe8abd640 1 -- 172.21.15.64:0/3297993521 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7effd4001c00 con 0x7effe413f710 2024-03-25T18:10:35.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.299+0000 7effe8abd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:35.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.299+0000 7effe8abd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:35.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.299+0000 7effe8abd640 10 monclient: dump: 2024-03-25T18:10:35.300 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:35.300 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:35.300 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:35.300 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:35.300 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:35.301 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:35.301 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:35.301 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:35.301 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:35.301 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:35.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.299+0000 7effebd23640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:35.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.299+0000 7effebd23640 4 set_mon_vals no callback set 2024-03-25T18:10:35.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.300+0000 7effebd23640 10 monclient: shutdown 2024-03-25T18:10:35.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.300+0000 7effebd23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:35.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.300+0000 7effebd23640 1 -- 172.21.15.64:0/3297993521 >> v1:172.21.15.86:6789/0 conn(0x7effe413f710 legacy=0x7effe41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:35.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.300+0000 7effebd23640 1 -- 172.21.15.64:0/3297993521 shutdown_connections 2024-03-25T18:10:35.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.300+0000 7effebd23640 1 -- 172.21.15.64:0/3297993521 >> 172.21.15.64:0/3297993521 conn(0x7effe40b7ea0 msgr2=0x7effe40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:35.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.300+0000 7effebd23640 1 -- 172.21.15.64:0/3297993521 shutdown_connections 2024-03-25T18:10:35.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.300+0000 7effebd23640 1 -- 172.21.15.64:0/3297993521 wait complete. 2024-03-25T18:10:35.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.301+0000 7effebd23640 10 monclient: build_initial_monmap 2024-03-25T18:10:35.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.301+0000 7effebd23640 10 monclient: monmap: 2024-03-25T18:10:35.303 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:10:35.303 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:35.303 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:10:35.303 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:10:35.303 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:35.303 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:10:35.303 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:10:35.303 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:10:35.303 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:10:35.303 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:35.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.301+0000 7effebd23640 1 librados: starting msgr at 2024-03-25T18:10:35.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.301+0000 7effebd23640 1 librados: starting objecter 2024-03-25T18:10:35.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.302+0000 7effebd23640 1 Processor -- start 2024-03-25T18:10:35.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.302+0000 7effebd23640 1 -- start start 2024-03-25T18:10:35.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.302+0000 7effebd23640 1 librados: setting wanted keys 2024-03-25T18:10:35.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.302+0000 7effebd23640 1 librados: calling monclient init 2024-03-25T18:10:35.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.302+0000 7effebd23640 10 monclient: init 2024-03-25T18:10:35.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.302+0000 7effebd23640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:35.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.302+0000 7effebd23640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:10:35.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.302+0000 7effebd23640 10 monclient(hunting): picked mon.noname-b con 0x7effe40a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:10:35.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.302+0000 7effebd23640 10 monclient(hunting): picked mon.noname-c con 0x7effe413f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:10:35.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.303+0000 7effebd23640 10 monclient(hunting): picked mon.noname-a con 0x7effe41430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:10:35.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.303+0000 7effebd23640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe41370a0 con 0x7effe41430c0 2024-03-25T18:10:35.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.303+0000 7effebd23640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe4135090 con 0x7effe40a98f0 2024-03-25T18:10:35.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.303+0000 7effebd23640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe4135220 con 0x7effe413f710 2024-03-25T18:10:35.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.303+0000 7effebd23640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:35.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.303+0000 7effebd23640 10 monclient(hunting): authenticate will time out at 7021.541637s 2024-03-25T18:10:35.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.313+0000 7effe92be640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7effe413f710 0x7effe41d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:47280/0 (socket says 172.21.15.64:47280) 2024-03-25T18:10:35.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.313+0000 7effe92be640 1 -- 172.21.15.64:0/1547645377 learned_addr learned my addr 172.21.15.64:0/1547645377 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:10:35.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.313+0000 7effda7fc640 10 client.?.objecter ms_handle_connect 0x7effe41430c0 2024-03-25T18:10:35.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.313+0000 7effda7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:35.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.315+0000 7effda7fc640 10 client.?.objecter ms_handle_connect 0x7effe413f710 2024-03-25T18:10:35.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.315+0000 7effda7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:35.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.317+0000 7effda7fc640 10 client.?.objecter ms_handle_connect 0x7effe40a98f0 2024-03-25T18:10:35.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.317+0000 7effda7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:35.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.321+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008120182 0 0) 0x7effe41370a0 con 0x7effe41430c0 2024-03-25T18:10:35.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.321+0000 7effda7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:35.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.321+0000 7effda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:35.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.321+0000 7effda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:35.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.321+0000 7effda7fc640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:10:35.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.321+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effc8004eb0 con 0x7effe41430c0 2024-03-25T18:10:35.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.322+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198250081 0 0) 0x7effe4135220 con 0x7effe413f710 2024-03-25T18:10:35.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.322+0000 7effda7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:35.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.322+0000 7effda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:35.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.322+0000 7effda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:35.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.322+0000 7effda7fc640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:10:35.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.322+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effe41370a0 con 0x7effe413f710 2024-03-25T18:10:35.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.332+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482315284 0 0) 0x7effe4135090 con 0x7effe40a98f0 2024-03-25T18:10:35.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.332+0000 7effda7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:35.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.332+0000 7effda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:35.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.332+0000 7effda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:35.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.332+0000 7effda7fc640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:10:35.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.332+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effe4135220 con 0x7effe40a98f0 2024-03-25T18:10:35.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.337+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1511240339 0 0) 0x7effe41370a0 con 0x7effe413f710 2024-03-25T18:10:35.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.337+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effe4135090 con 0x7effe413f710 2024-03-25T18:10:35.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.337+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7effd40015d0 con 0x7effe413f710 2024-03-25T18:10:35.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.337+0000 7effda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:35.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.337+0000 7effda7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:35.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.337+0000 7effda7fc640 10 monclient(hunting): dump: 2024-03-25T18:10:35.338 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:35.339 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:35.339 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:35.339 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:35.339 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:35.339 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:35.339 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:35.339 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:35.339 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:35.339 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:35.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.337+0000 7effda7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:35.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.341+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 699033761 0 0) 0x7effc8004eb0 con 0x7effe41430c0 2024-03-25T18:10:35.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.341+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effe41370a0 con 0x7effe41430c0 2024-03-25T18:10:35.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.341+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7effdc001b90 con 0x7effe41430c0 2024-03-25T18:10:35.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.341+0000 7effda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:35.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.341+0000 7effda7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:35.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.341+0000 7effda7fc640 10 monclient(hunting): dump: 2024-03-25T18:10:35.342 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:35.342 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:35.342 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:35.342 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:35.342 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:35.342 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:35.342 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:35.342 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:35.342 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:35.342 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:35.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.342+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 805180525 0 0) 0x7effe4135220 con 0x7effe40a98f0 2024-03-25T18:10:35.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.342+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effc8004eb0 con 0x7effe40a98f0 2024-03-25T18:10:35.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.342+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7effe0001830 con 0x7effe40a98f0 2024-03-25T18:10:35.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.342+0000 7effda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:35.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.342+0000 7effda7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:35.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.342+0000 7effda7fc640 10 monclient(hunting): dump: 2024-03-25T18:10:35.343 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:35.343 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:35.344 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:35.344 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:35.344 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:35.344 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:35.344 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:35.344 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:35.344 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:35.344 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:35.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3945780186 0 0) 0x7effe41370a0 con 0x7effe41430c0 2024-03-25T18:10:35.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 >> v1:172.21.15.64:6790/0 conn(0x7effe413f710 legacy=0x7effe41d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:35.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 >> v1:172.21.15.86:6789/0 conn(0x7effe40a98f0 legacy=0x7effe41dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:35.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effda7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:10:35.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effda7fc640 1 monclient: found mon.a 2024-03-25T18:10:35.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:35.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effda7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:10:35.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe41d5000 con 0x7effe41430c0 2024-03-25T18:10:35.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effda7fc640 10 monclient: _finish_auth 0 2024-03-25T18:10:35.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effda7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:10:35.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:35.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effebd23640 5 monclient: authenticate success, global_id 4269 2024-03-25T18:10:35.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effebd23640 10 monclient: _renew_subs 2024-03-25T18:10:35.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effebd23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:10:35.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effebd23640 1 -- 172.21.15.64:0/1547645377 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe41d52b0 con 0x7effe41430c0 2024-03-25T18:10:35.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effebd23640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:35.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effebd23640 10 monclient: _renew_subs 2024-03-25T18:10:35.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effebd23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:10:35.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effebd23640 1 -- 172.21.15.64:0/1547645377 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7effe4135910 con 0x7effe41430c0 2024-03-25T18:10:35.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.353+0000 7effebd23640 1 librados: init done 2024-03-25T18:10:35.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.354+0000 7effebd23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:35.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.354+0000 7effebd23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:35.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.354+0000 7effebd23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:10:35.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.354+0000 7effebd23640 1 -- 172.21.15.64:0/1547645377 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe40627b0 con 0x7effe41430c0 2024-03-25T18:10:35.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.681+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effdc0020c0 con 0x7effe41430c0 2024-03-25T18:10:35.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.681+0000 7effda7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:35.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.681+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7effdc003b10 con 0x7effe41430c0 2024-03-25T18:10:35.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.681+0000 7effda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:35.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.681+0000 7effda7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:35.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.681+0000 7effda7fc640 10 monclient: dump: 2024-03-25T18:10:35.682 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:35.682 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:35.682 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:35.682 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:35.682 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:35.683 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:35.683 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:35.683 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:35.683 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:35.683 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:35.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.681+0000 7effdbfff640 4 set_mon_vals no callback set 2024-03-25T18:10:35.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.682+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7effdc003d30 con 0x7effe41430c0 2024-03-25T18:10:35.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.683+0000 7effe9abf640 10 monclient: get_auth_request con 0x7effc806e0c0 auth_method 0 2024-03-25T18:10:35.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.683+0000 7effda7fc640 10 client.4269.objecter ms_handle_connect 0x7effc806e0c0 2024-03-25T18:10:35.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.691+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7effdc0538c0 con 0x7effe41430c0 2024-03-25T18:10:35.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.691+0000 7effda7fc640 10 client.4269.objecter ms_dispatch 0x7effe4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:35.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.691+0000 7effda7fc640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:35.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.691+0000 7effda7fc640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:35.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.691+0000 7effda7fc640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-25T18:10:35.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.691+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7effdc04f3d0 con 0x7effe41430c0 2024-03-25T18:10:35.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.691+0000 7effda7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:35.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.691+0000 7effda7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:35.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.925+0000 7effebd23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:35.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.925+0000 7effebd23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:35.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.925+0000 7effebd23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:10:35.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:35.925+0000 7effebd23640 1 -- 172.21.15.64:0/1547645377 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effe414f240 con 0x7effe41430c0 2024-03-25T18:10:36.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:36.051+0000 7effda7fc640 1 -- 172.21.15.64:0/1547645377 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7effdc051300 con 0x7effe41430c0 2024-03-25T18:10:36.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:36.052+0000 7effda7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:36.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:36.052+0000 7effda7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:36.053 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:10:36.053 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:36.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:36.054+0000 7effebd23640 10 librados: watch_flush enter 2024-03-25T18:10:36.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:36.054+0000 7effebd23640 10 librados: watch_flush exit 2024-03-25T18:10:36.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:36.054+0000 7effebd23640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-25T18:10:36.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:36.054+0000 7effebd23640 10 client.4269.objecter successfully canceled tick 2024-03-25T18:10:36.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:36.054+0000 7effebd23640 1 -- 172.21.15.64:0/1547645377 >> v1:172.21.15.64:6800/2915543638 conn(0x7effc806e0c0 legacy=0x7effc8070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:36.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:36.054+0000 7effebd23640 10 monclient: shutdown 2024-03-25T18:10:36.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:36.054+0000 7effebd23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:36.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:36.054+0000 7effebd23640 1 -- 172.21.15.64:0/1547645377 >> v1:172.21.15.64:6789/0 conn(0x7effe41430c0 legacy=0x7effe41d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:36.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:36.054+0000 7effebd23640 1 -- 172.21.15.64:0/1547645377 shutdown_connections 2024-03-25T18:10:36.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:36.054+0000 7effebd23640 1 -- 172.21.15.64:0/1547645377 >> 172.21.15.64:0/1547645377 conn(0x7effe40b7ea0 msgr2=0x7effe4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:36.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:36.054+0000 7effebd23640 1 -- 172.21.15.64:0/1547645377 shutdown_connections 2024-03-25T18:10:36.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:36.054+0000 7effebd23640 1 -- 172.21.15.64:0/1547645377 wait complete. 2024-03-25T18:10:36.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:36.054+0000 7effebd23640 1 librados: shutdown 2024-03-25T18:10:36.065 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:42.872 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:42.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.973+0000 7faeb42eb640 10 monclient: get_monmap_and_config 2024-03-25T18:10:42.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.973+0000 7faeb42eb640 10 monclient: build_initial_monmap 2024-03-25T18:10:42.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.973+0000 7faeb42eb640 10 monclient: monmap: 2024-03-25T18:10:42.974 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:10:42.974 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:42.974 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:10:42.974621+0000 2024-03-25T18:10:42.974 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:10:42.974621+0000 2024-03-25T18:10:42.974 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:42.975 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:42.975 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:10:42.975 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:10:42.975 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:10:42.975 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:42.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.974+0000 7faeb42eb640 1 Processor -- start 2024-03-25T18:10:42.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.974+0000 7faeb42eb640 1 -- start start 2024-03-25T18:10:42.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.974+0000 7faeb42eb640 10 monclient: init 2024-03-25T18:10:42.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.975+0000 7faeb42eb640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:42.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.975+0000 7faeb42eb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:10:42.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.975+0000 7faeb42eb640 10 monclient(hunting): picked mon.noname-a con 0x7faeac0b4d60 addr v1:172.21.15.64:6789/0 2024-03-25T18:10:42.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.975+0000 7faeb42eb640 10 monclient(hunting): picked mon.noname-b con 0x7faeac0b1730 addr v1:172.21.15.64:6790/0 2024-03-25T18:10:42.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.975+0000 7faeb42eb640 10 monclient(hunting): picked mon.noname-c con 0x7faeac14be90 addr v1:172.21.15.86:6789/0 2024-03-25T18:10:42.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.975+0000 7faeb42eb640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeac0af940 con 0x7faeac0b4d60 2024-03-25T18:10:42.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.975+0000 7faeb42eb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeac0b0a70 con 0x7faeac14be90 2024-03-25T18:10:42.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.975+0000 7faeb42eb640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeac0b0d50 con 0x7faeac0b1730 2024-03-25T18:10:42.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.975+0000 7faeb42eb640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:42.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.975+0000 7faeb42eb640 10 monclient(hunting): authenticate will time out at 7029.213643s 2024-03-25T18:10:42.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.975+0000 7faeb2087640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7faeac0b4d60 0x7faeac13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35888/0 (socket says 172.21.15.64:35888) 2024-03-25T18:10:42.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.975+0000 7faeb2087640 1 -- 172.21.15.64:0/3401753580 learned_addr learned my addr 172.21.15.64:0/3401753580 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:10:42.995 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.995+0000 7faeb1085640 1 -- 172.21.15.64:0/3401753580 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434615059 0 0) 0x7faeac0b0a70 con 0x7faeac14be90 2024-03-25T18:10:42.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.995+0000 7faeb1085640 10 monclient(hunting): _negotiate 2024-03-25T18:10:42.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.995+0000 7faeb1085640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:42.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.995+0000 7faeb1085640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:42.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.995+0000 7faeb1085640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:10:42.996 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:42.995+0000 7faeb1085640 1 -- 172.21.15.64:0/3401753580 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae98003520 con 0x7faeac14be90 2024-03-25T18:10:43.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.015+0000 7faeb1085640 1 -- 172.21.15.64:0/3401753580 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4232170401 0 0) 0x7fae98003520 con 0x7faeac14be90 2024-03-25T18:10:43.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.016+0000 7faeb1085640 1 -- 172.21.15.64:0/3401753580 >> v1:172.21.15.64:6790/0 conn(0x7faeac0b1730 legacy=0x7faeac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:43.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.016+0000 7faeb1085640 1 -- 172.21.15.64:0/3401753580 >> v1:172.21.15.64:6789/0 conn(0x7faeac0b4d60 legacy=0x7faeac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:43.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.016+0000 7faeb1085640 10 monclient: _finish_hunting 0 2024-03-25T18:10:43.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.016+0000 7faeb1085640 1 monclient: found mon.noname-c 2024-03-25T18:10:43.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.016+0000 7faeb1085640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:43.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.016+0000 7faeb1085640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:10:43.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.016+0000 7faeb1085640 1 -- 172.21.15.64:0/3401753580 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeac0b1030 con 0x7faeac14be90 2024-03-25T18:10:43.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.016+0000 7faeb1085640 10 monclient: _finish_auth 0 2024-03-25T18:10:43.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.016+0000 7faeb1085640 10 monclient: _check_auth_tickets 2024-03-25T18:10:43.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.016+0000 7faeb1085640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:43.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.016+0000 7faeb1085640 1 -- 172.21.15.64:0/3401753580 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7faea4001a10 con 0x7faeac14be90 2024-03-25T18:10:43.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.016+0000 7faeb1085640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:43.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.016+0000 7faeb1085640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:43.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.016+0000 7faeb1085640 10 monclient: dump: 2024-03-25T18:10:43.017 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:43.017 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:43.017 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:43.017 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:43.018 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:43.018 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:43.018 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:43.018 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:43.018 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:43.018 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:43.018 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.016+0000 7faeb42eb640 5 monclient: authenticate success, global_id 4261 2024-03-25T18:10:43.018 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.016+0000 7faeb42eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:43.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.041+0000 7faeb1085640 1 -- 172.21.15.64:0/3401753580 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faea4001b70 con 0x7faeac14be90 2024-03-25T18:10:43.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.041+0000 7faeb1085640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:43.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.041+0000 7faeb1085640 1 -- 172.21.15.64:0/3401753580 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7faea4001fc0 con 0x7faeac14be90 2024-03-25T18:10:43.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.041+0000 7faeb42eb640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:43.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.041+0000 7faeb42eb640 4 set_mon_vals no callback set 2024-03-25T18:10:43.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.041+0000 7faeb1085640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:43.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.041+0000 7faeb1085640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:43.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.041+0000 7faeb1085640 10 monclient: dump: 2024-03-25T18:10:43.042 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:43.042 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:43.042 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:43.042 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:43.042 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:43.042 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:43.042 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:43.042 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:43.042 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:43.042 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:43.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.041+0000 7faeb42eb640 10 monclient: shutdown 2024-03-25T18:10:43.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.041+0000 7faeb42eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:43.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.041+0000 7faeb42eb640 1 -- 172.21.15.64:0/3401753580 >> v1:172.21.15.86:6789/0 conn(0x7faeac14be90 legacy=0x7faeac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:43.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.042+0000 7faeb42eb640 1 -- 172.21.15.64:0/3401753580 shutdown_connections 2024-03-25T18:10:43.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.042+0000 7faeb42eb640 1 -- 172.21.15.64:0/3401753580 >> 172.21.15.64:0/3401753580 conn(0x7faeac0617a0 msgr2=0x7faeac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:43.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.042+0000 7faeb42eb640 1 -- 172.21.15.64:0/3401753580 shutdown_connections 2024-03-25T18:10:43.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.042+0000 7faeb42eb640 1 -- 172.21.15.64:0/3401753580 wait complete. 2024-03-25T18:10:43.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.042+0000 7faeb42eb640 10 monclient: build_initial_monmap 2024-03-25T18:10:43.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.042+0000 7faeb42eb640 10 monclient: monmap: 2024-03-25T18:10:43.045 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:10:43.045 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:43.045 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:10:43.045 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:10:43.045 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:43.045 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:10:43.045 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:10:43.045 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:10:43.045 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:10:43.045 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:43.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.043+0000 7faeb42eb640 1 librados: starting msgr at 2024-03-25T18:10:43.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.043+0000 7faeb42eb640 1 librados: starting objecter 2024-03-25T18:10:43.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.043+0000 7faeb42eb640 1 Processor -- start 2024-03-25T18:10:43.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.043+0000 7faeb42eb640 1 -- start start 2024-03-25T18:10:43.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.043+0000 7faeb42eb640 1 librados: setting wanted keys 2024-03-25T18:10:43.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.043+0000 7faeb42eb640 1 librados: calling monclient init 2024-03-25T18:10:43.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.043+0000 7faeb42eb640 10 monclient: init 2024-03-25T18:10:43.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.043+0000 7faeb42eb640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:43.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.043+0000 7faeb42eb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:10:43.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.043+0000 7faeb42eb640 10 monclient(hunting): picked mon.noname-a con 0x7faeac0b1730 addr v1:172.21.15.64:6789/0 2024-03-25T18:10:43.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.043+0000 7faeb42eb640 10 monclient(hunting): picked mon.noname-b con 0x7faeac0b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:10:43.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.043+0000 7faeb42eb640 10 monclient(hunting): picked mon.noname-c con 0x7faeac14be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:10:43.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.043+0000 7faeb42eb640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeac138640 con 0x7faeac0b1730 2024-03-25T18:10:43.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.043+0000 7faeb42eb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeac139800 con 0x7faeac0b4d60 2024-03-25T18:10:43.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.043+0000 7faeb42eb640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeac132620 con 0x7faeac14be90 2024-03-25T18:10:43.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.043+0000 7faeb42eb640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:43.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.043+0000 7faeb42eb640 10 monclient(hunting): authenticate will time out at 7029.282274s 2024-03-25T18:10:43.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.062+0000 7faeb1886640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7faeac0b4d60 0x7faeac13b040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:38616/0 (socket says 172.21.15.64:38616) 2024-03-25T18:10:43.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.062+0000 7faeb1886640 1 -- 172.21.15.64:0/231975722 learned_addr learned my addr 172.21.15.64:0/231975722 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:10:43.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.063+0000 7faea2ffd640 10 client.?.objecter ms_handle_connect 0x7faeac14be90 2024-03-25T18:10:43.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.063+0000 7faea2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:43.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.063+0000 7faea2ffd640 10 client.?.objecter ms_handle_connect 0x7faeac0b4d60 2024-03-25T18:10:43.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.063+0000 7faea2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:43.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.070+0000 7faea2ffd640 1 -- 172.21.15.64:0/231975722 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021050941 0 0) 0x7faeac132620 con 0x7faeac14be90 2024-03-25T18:10:43.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.070+0000 7faea2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:43.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.070+0000 7faea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:43.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.070+0000 7faea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:43.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.070+0000 7faea2ffd640 10 monclient(hunting): my global_id is 4274 2024-03-25T18:10:43.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.070+0000 7faea2ffd640 1 -- 172.21.15.64:0/231975722 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae90004eb0 con 0x7faeac14be90 2024-03-25T18:10:43.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.074+0000 7faea2ffd640 1 -- 172.21.15.64:0/231975722 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164912952 0 0) 0x7faeac139800 con 0x7faeac0b4d60 2024-03-25T18:10:43.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.074+0000 7faea2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:43.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.074+0000 7faea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:43.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.074+0000 7faea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:43.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.074+0000 7faea2ffd640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:10:43.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.074+0000 7faea2ffd640 1 -- 172.21.15.64:0/231975722 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeac132620 con 0x7faeac0b4d60 2024-03-25T18:10:43.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.080+0000 7faea2ffd640 1 -- 172.21.15.64:0/231975722 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2521886218 0 0) 0x7fae90004eb0 con 0x7faeac14be90 2024-03-25T18:10:43.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.080+0000 7faea2ffd640 1 -- 172.21.15.64:0/231975722 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faeac139800 con 0x7faeac14be90 2024-03-25T18:10:43.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.080+0000 7faea2ffd640 1 -- 172.21.15.64:0/231975722 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7faea4001e80 con 0x7faeac14be90 2024-03-25T18:10:43.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.080+0000 7faea2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:43.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.080+0000 7faea2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:43.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.080+0000 7faea2ffd640 10 monclient(hunting): dump: 2024-03-25T18:10:43.081 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:43.081 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:43.081 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:43.081 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:43.081 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:43.081 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:43.081 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:43.081 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:43.081 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:43.082 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:43.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.080+0000 7faea2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:43.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.082+0000 7faea2ffd640 10 client.?.objecter ms_handle_connect 0x7faeac0b1730 2024-03-25T18:10:43.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.082+0000 7faea2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faea2ffd640 1 -- 172.21.15.64:0/231975722 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3766226926 0 0) 0x7faeac139800 con 0x7faeac14be90 2024-03-25T18:10:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faea2ffd640 1 -- 172.21.15.64:0/231975722 >> v1:172.21.15.86:6789/0 conn(0x7faeac0b4d60 legacy=0x7faeac13b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faea2ffd640 1 -- 172.21.15.64:0/231975722 >> v1:172.21.15.64:6789/0 conn(0x7faeac0b1730 legacy=0x7faeac131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faea2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:10:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faea2ffd640 1 monclient: found mon.c 2024-03-25T18:10:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faea2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faea2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:10:43.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faea2ffd640 1 -- 172.21.15.64:0/231975722 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeac1327b0 con 0x7faeac14be90 2024-03-25T18:10:43.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faea2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:10:43.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faea2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:10:43.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faea2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:43.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faeb42eb640 5 monclient: authenticate success, global_id 4274 2024-03-25T18:10:43.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faeb42eb640 10 monclient: _renew_subs 2024-03-25T18:10:43.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faeb42eb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:10:43.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faeb42eb640 1 -- 172.21.15.64:0/231975722 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeac138dd0 con 0x7faeac14be90 2024-03-25T18:10:43.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faeb42eb640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:43.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faeb42eb640 10 monclient: _renew_subs 2024-03-25T18:10:43.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faeb42eb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:10:43.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faeb42eb640 1 -- 172.21.15.64:0/231975722 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faeac138270 con 0x7faeac14be90 2024-03-25T18:10:43.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.088+0000 7faeb42eb640 1 librados: init done 2024-03-25T18:10:43.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.089+0000 7faeb42eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:43.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.089+0000 7faeb42eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:43.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.089+0000 7faeb42eb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:10:43.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.089+0000 7faeb42eb640 1 -- 172.21.15.64:0/231975722 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeac0b2270 con 0x7faeac14be90 2024-03-25T18:10:43.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.311+0000 7faea2ffd640 1 -- 172.21.15.64:0/231975722 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faea4003c20 con 0x7faeac14be90 2024-03-25T18:10:43.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.311+0000 7faea2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:43.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.311+0000 7faeb0884640 4 set_mon_vals no callback set 2024-03-25T18:10:43.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.411+0000 7faea2ffd640 1 -- 172.21.15.64:0/231975722 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7faea4002100 con 0x7faeac14be90 2024-03-25T18:10:43.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.411+0000 7faea2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:43.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.411+0000 7faea2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:43.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.411+0000 7faea2ffd640 10 monclient: dump: 2024-03-25T18:10:43.412 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:43.412 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:43.412 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:43.412 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:43.412 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:43.412 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:43.412 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:43.412 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:43.412 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:43.413 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:43.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.412+0000 7faea2ffd640 1 -- 172.21.15.64:0/231975722 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7faea4019da0 con 0x7faeac14be90 2024-03-25T18:10:43.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.412+0000 7faeb2087640 10 monclient: get_auth_request con 0x7fae9006dd00 auth_method 0 2024-03-25T18:10:43.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.413+0000 7faea2ffd640 10 client.4274.objecter ms_handle_connect 0x7fae9006dd00 2024-03-25T18:10:43.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.419+0000 7faea2ffd640 1 -- 172.21.15.64:0/231975722 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7faea40539b0 con 0x7faeac14be90 2024-03-25T18:10:43.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.419+0000 7faea2ffd640 10 client.4274.objecter ms_dispatch 0x7faeac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:43.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.419+0000 7faea2ffd640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:43.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.419+0000 7faea2ffd640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:43.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.419+0000 7faea2ffd640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-25T18:10:43.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.419+0000 7faea2ffd640 1 -- 172.21.15.64:0/231975722 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faea404f2f0 con 0x7faeac14be90 2024-03-25T18:10:43.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.419+0000 7faea2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:43.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.419+0000 7faea2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:43.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.668+0000 7faeb42eb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:43.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.668+0000 7faeb42eb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:43.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.668+0000 7faeb42eb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:10:43.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.668+0000 7faeb42eb640 1 -- 172.21.15.64:0/231975722 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faeac1387d0 con 0x7faeac14be90 2024-03-25T18:10:43.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.692+0000 7faea2ffd640 1 -- 172.21.15.64:0/231975722 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7faea40513f0 con 0x7faeac14be90 2024-03-25T18:10:43.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.692+0000 7faea2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:43.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.692+0000 7faea2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:43.693 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:10:43.693 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:43.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.694+0000 7faeb42eb640 10 librados: watch_flush enter 2024-03-25T18:10:43.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.694+0000 7faeb42eb640 10 librados: watch_flush exit 2024-03-25T18:10:43.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.694+0000 7faeb42eb640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-25T18:10:43.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.694+0000 7faeb42eb640 10 client.4274.objecter successfully canceled tick 2024-03-25T18:10:43.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.694+0000 7faeb42eb640 1 -- 172.21.15.64:0/231975722 >> v1:172.21.15.64:6800/2915543638 conn(0x7fae9006dd00 legacy=0x7fae900701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:43.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.694+0000 7faeb42eb640 10 monclient: shutdown 2024-03-25T18:10:43.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.694+0000 7faeb42eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:43.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.694+0000 7faeb42eb640 1 -- 172.21.15.64:0/231975722 >> v1:172.21.15.64:6790/0 conn(0x7faeac14be90 legacy=0x7faeac13b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:43.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.694+0000 7faeb42eb640 1 -- 172.21.15.64:0/231975722 shutdown_connections 2024-03-25T18:10:43.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.694+0000 7faeb42eb640 1 -- 172.21.15.64:0/231975722 >> 172.21.15.64:0/231975722 conn(0x7faeac0617a0 msgr2=0x7faeac13c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:43.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.694+0000 7faeb42eb640 1 -- 172.21.15.64:0/231975722 shutdown_connections 2024-03-25T18:10:43.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.694+0000 7faeb42eb640 1 -- 172.21.15.64:0/231975722 wait complete. 2024-03-25T18:10:43.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:43.694+0000 7faeb42eb640 1 librados: shutdown 2024-03-25T18:10:43.706 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:50.512 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:50.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.629+0000 7fa6c5406640 10 monclient: get_monmap_and_config 2024-03-25T18:10:50.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.629+0000 7fa6c5406640 10 monclient: build_initial_monmap 2024-03-25T18:10:50.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.629+0000 7fa6c5406640 10 monclient: monmap: 2024-03-25T18:10:50.630 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:10:50.630 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:50.630 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:10:50.630039+0000 2024-03-25T18:10:50.630 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:10:50.630039+0000 2024-03-25T18:10:50.630 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:50.630 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:50.630 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:10:50.631 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:10:50.631 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:10:50.631 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:50.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.629+0000 7fa6c5406640 1 Processor -- start 2024-03-25T18:10:50.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.629+0000 7fa6c5406640 1 -- start start 2024-03-25T18:10:50.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.629+0000 7fa6c5406640 10 monclient: init 2024-03-25T18:10:50.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.630+0000 7fa6c5406640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:50.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.630+0000 7fa6c5406640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:50.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.630+0000 7fa6c5406640 10 monclient(hunting): picked mon.noname-c con 0x7fa6c013f990 addr v1:172.21.15.86:6789/0 2024-03-25T18:10:50.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.630+0000 7fa6c5406640 10 monclient(hunting): picked mon.noname-a con 0x7fa6c013fd70 addr v1:172.21.15.64:6789/0 2024-03-25T18:10:50.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.630+0000 7fa6c5406640 10 monclient(hunting): picked mon.noname-b con 0x7fa6c00afa60 addr v1:172.21.15.64:6790/0 2024-03-25T18:10:50.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.630+0000 7fa6c5406640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c00afeb0 con 0x7fa6c013fd70 2024-03-25T18:10:50.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.630+0000 7fa6c5406640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c00b0100 con 0x7fa6c013f990 2024-03-25T18:10:50.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.630+0000 7fa6c5406640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c00b1150 con 0x7fa6c00afa60 2024-03-25T18:10:50.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.630+0000 7fa6c5406640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:50.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.630+0000 7fa6c5406640 10 monclient(hunting): authenticate will time out at 7036.869399s 2024-03-25T18:10:50.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.639+0000 7fa6be7fc640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fa6c013fd70 0x7fa6c0145850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:56574/0 (socket says 172.21.15.64:56574) 2024-03-25T18:10:50.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.639+0000 7fa6be7fc640 1 -- 172.21.15.64:0/3558723629 learned_addr learned my addr 172.21.15.64:0/3558723629 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:10:50.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.654+0000 7fa6bdffb640 1 -- 172.21.15.64:0/3558723629 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112404911 0 0) 0x7fa6c00afeb0 con 0x7fa6c013fd70 2024-03-25T18:10:50.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.655+0000 7fa6bdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:50.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.655+0000 7fa6bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:50.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.655+0000 7fa6bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:50.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.655+0000 7fa6bdffb640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:10:50.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.655+0000 7fa6bdffb640 1 -- 172.21.15.64:0/3558723629 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a8003520 con 0x7fa6c013fd70 2024-03-25T18:10:50.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.660+0000 7fa6bdffb640 1 -- 172.21.15.64:0/3558723629 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043610002 0 0) 0x7fa6c00b1150 con 0x7fa6c00afa60 2024-03-25T18:10:50.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.660+0000 7fa6bdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:50.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.660+0000 7fa6bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:50.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.660+0000 7fa6bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:50.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.660+0000 7fa6bdffb640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:10:50.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.660+0000 7fa6bdffb640 1 -- 172.21.15.64:0/3558723629 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c00afeb0 con 0x7fa6c00afa60 2024-03-25T18:10:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.663+0000 7fa6bdffb640 1 -- 172.21.15.64:0/3558723629 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672702466 0 0) 0x7fa6c00b0100 con 0x7fa6c013f990 2024-03-25T18:10:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.663+0000 7fa6bdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.663+0000 7fa6bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.663+0000 7fa6bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:50.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.663+0000 7fa6bdffb640 10 monclient(hunting): my global_id is 4267 2024-03-25T18:10:50.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.663+0000 7fa6bdffb640 1 -- 172.21.15.64:0/3558723629 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c00b1150 con 0x7fa6c013f990 2024-03-25T18:10:50.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.674+0000 7fa6bdffb640 1 -- 172.21.15.64:0/3558723629 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2257509946 0 0) 0x7fa6c00afeb0 con 0x7fa6c00afa60 2024-03-25T18:10:50.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.674+0000 7fa6bdffb640 1 -- 172.21.15.64:0/3558723629 >> v1:172.21.15.86:6789/0 conn(0x7fa6c013f990 legacy=0x7fa6c006ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:50.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.674+0000 7fa6bdffb640 1 -- 172.21.15.64:0/3558723629 >> v1:172.21.15.64:6789/0 conn(0x7fa6c013fd70 legacy=0x7fa6c0145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:50.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.674+0000 7fa6bdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:10:50.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.674+0000 7fa6bdffb640 1 monclient: found mon.noname-b 2024-03-25T18:10:50.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.674+0000 7fa6bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:50.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.674+0000 7fa6bdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:10:50.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.674+0000 7fa6bdffb640 1 -- 172.21.15.64:0/3558723629 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c00aede0 con 0x7fa6c00afa60 2024-03-25T18:10:50.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.674+0000 7fa6bdffb640 10 monclient: _finish_auth 0 2024-03-25T18:10:50.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.674+0000 7fa6bdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:10:50.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.674+0000 7fa6bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:50.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.674+0000 7fa6bdffb640 1 -- 172.21.15.64:0/3558723629 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fa6b80019f0 con 0x7fa6c00afa60 2024-03-25T18:10:50.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.674+0000 7fa6bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:50.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.674+0000 7fa6bdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:50.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.674+0000 7fa6bdffb640 10 monclient: dump: 2024-03-25T18:10:50.677 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:50.677 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:50.677 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:50.677 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:50.677 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:50.677 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:50.678 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:50.678 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:50.678 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:50.678 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:50.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.675+0000 7fa6c5406640 5 monclient: authenticate success, global_id 4277 2024-03-25T18:10:50.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.675+0000 7fa6c5406640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:50.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.689+0000 7fa6bdffb640 1 -- 172.21.15.64:0/3558723629 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6b8001b50 con 0x7fa6c00afa60 2024-03-25T18:10:50.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.689+0000 7fa6bdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:50.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.689+0000 7fa6bdffb640 1 -- 172.21.15.64:0/3558723629 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fa6b8001f60 con 0x7fa6c00afa60 2024-03-25T18:10:50.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.689+0000 7fa6bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:50.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.689+0000 7fa6bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:50.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.689+0000 7fa6bdffb640 10 monclient: dump: 2024-03-25T18:10:50.690 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:50.690 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:50.690 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:50.690 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:50.690 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:50.690 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:50.691 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:50.691 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:50.691 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:50.691 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:50.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.689+0000 7fa6c5406640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:50.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.689+0000 7fa6c5406640 4 set_mon_vals no callback set 2024-03-25T18:10:50.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.689+0000 7fa6c5406640 10 monclient: shutdown 2024-03-25T18:10:50.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.689+0000 7fa6c5406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:50.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.689+0000 7fa6c5406640 1 -- 172.21.15.64:0/3558723629 >> v1:172.21.15.64:6790/0 conn(0x7fa6c00afa60 legacy=0x7fa6c014dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:50.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.690+0000 7fa6c5406640 1 -- 172.21.15.64:0/3558723629 shutdown_connections 2024-03-25T18:10:50.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.690+0000 7fa6c5406640 1 -- 172.21.15.64:0/3558723629 >> 172.21.15.64:0/3558723629 conn(0x7fa6c00b4b80 msgr2=0x7fa6c00b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:50.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.690+0000 7fa6c5406640 1 -- 172.21.15.64:0/3558723629 shutdown_connections 2024-03-25T18:10:50.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.690+0000 7fa6c5406640 1 -- 172.21.15.64:0/3558723629 wait complete. 2024-03-25T18:10:50.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.691+0000 7fa6c5406640 10 monclient: build_initial_monmap 2024-03-25T18:10:50.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.691+0000 7fa6c5406640 10 monclient: monmap: 2024-03-25T18:10:50.695 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:10:50.695 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:50.695 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:10:50.695 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:10:50.695 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:50.695 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:10:50.695 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:10:50.696 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:10:50.696 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:10:50.696 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:50.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.691+0000 7fa6c5406640 1 librados: starting msgr at 2024-03-25T18:10:50.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.691+0000 7fa6c5406640 1 librados: starting objecter 2024-03-25T18:10:50.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.691+0000 7fa6c5406640 1 Processor -- start 2024-03-25T18:10:50.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.691+0000 7fa6c5406640 1 -- start start 2024-03-25T18:10:50.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.691+0000 7fa6c5406640 1 librados: setting wanted keys 2024-03-25T18:10:50.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.691+0000 7fa6c5406640 1 librados: calling monclient init 2024-03-25T18:10:50.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.691+0000 7fa6c5406640 10 monclient: init 2024-03-25T18:10:50.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.692+0000 7fa6c5406640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:50.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.692+0000 7fa6c5406640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:50.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.692+0000 7fa6c5406640 10 monclient(hunting): picked mon.noname-b con 0x7fa6c00afa60 addr v1:172.21.15.86:6789/0 2024-03-25T18:10:50.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.692+0000 7fa6c5406640 10 monclient(hunting): picked mon.noname-a con 0x7fa6c013f990 addr v1:172.21.15.64:6789/0 2024-03-25T18:10:50.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.692+0000 7fa6c5406640 10 monclient(hunting): picked mon.noname-c con 0x7fa6c013fd70 addr v1:172.21.15.64:6790/0 2024-03-25T18:10:50.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.692+0000 7fa6c5406640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c0120b60 con 0x7fa6c013f990 2024-03-25T18:10:50.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.692+0000 7fa6c5406640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c013e3f0 con 0x7fa6c00afa60 2024-03-25T18:10:50.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.692+0000 7fa6c5406640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c011f2d0 con 0x7fa6c013fd70 2024-03-25T18:10:50.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.692+0000 7fa6c5406640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:50.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.692+0000 7fa6c5406640 10 monclient(hunting): authenticate will time out at 7036.931430s 2024-03-25T18:10:50.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.695+0000 7fa6beffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa6c00afa60 0x7fa6c01da870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:38500/0 (socket says 172.21.15.64:38500) 2024-03-25T18:10:50.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.695+0000 7fa6beffd640 1 -- 172.21.15.64:0/2633585968 learned_addr learned my addr 172.21.15.64:0/2633585968 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:10:50.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.696+0000 7fa6a77fe640 10 client.?.objecter ms_handle_connect 0x7fa6c00afa60 2024-03-25T18:10:50.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.696+0000 7fa6a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:50.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.704+0000 7fa6a77fe640 10 client.?.objecter ms_handle_connect 0x7fa6c013fd70 2024-03-25T18:10:50.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.704+0000 7fa6a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:50.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.708+0000 7fa6a77fe640 1 -- 172.21.15.64:0/2633585968 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136336235 0 0) 0x7fa6c013e3f0 con 0x7fa6c00afa60 2024-03-25T18:10:50.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.708+0000 7fa6a77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:50.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.708+0000 7fa6a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:50.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.708+0000 7fa6a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:50.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.708+0000 7fa6a77fe640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:10:50.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.708+0000 7fa6a77fe640 1 -- 172.21.15.64:0/2633585968 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6ac004eb0 con 0x7fa6c00afa60 2024-03-25T18:10:50.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.717+0000 7fa6a77fe640 1 -- 172.21.15.64:0/2633585968 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940432087 0 0) 0x7fa6c011f2d0 con 0x7fa6c013fd70 2024-03-25T18:10:50.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.717+0000 7fa6a77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:50.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.717+0000 7fa6a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:50.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.717+0000 7fa6a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:50.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.717+0000 7fa6a77fe640 10 monclient(hunting): my global_id is 4280 2024-03-25T18:10:50.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.718+0000 7fa6a77fe640 1 -- 172.21.15.64:0/2633585968 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c013e3f0 con 0x7fa6c013fd70 2024-03-25T18:10:50.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.721+0000 7fa6a77fe640 1 -- 172.21.15.64:0/2633585968 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3924974034 0 0) 0x7fa6ac004eb0 con 0x7fa6c00afa60 2024-03-25T18:10:50.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.721+0000 7fa6a77fe640 1 -- 172.21.15.64:0/2633585968 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6c011f2d0 con 0x7fa6c00afa60 2024-03-25T18:10:50.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.721+0000 7fa6a77fe640 1 -- 172.21.15.64:0/2633585968 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fa6b0001a10 con 0x7fa6c00afa60 2024-03-25T18:10:50.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.721+0000 7fa6a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:50.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.721+0000 7fa6a77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:50.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.721+0000 7fa6a77fe640 10 monclient(hunting): dump: 2024-03-25T18:10:50.723 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:50.723 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:50.723 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:50.723 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:50.723 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:50.723 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:50.723 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:50.723 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:50.723 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:50.724 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:50.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.721+0000 7fa6a77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:50.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.724+0000 7fa6a77fe640 10 client.?.objecter ms_handle_connect 0x7fa6c013f990 2024-03-25T18:10:50.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.724+0000 7fa6a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:50.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.734+0000 7fa6a77fe640 1 -- 172.21.15.64:0/2633585968 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1509910312 0 0) 0x7fa6c011f2d0 con 0x7fa6c00afa60 2024-03-25T18:10:50.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.734+0000 7fa6a77fe640 1 -- 172.21.15.64:0/2633585968 >> v1:172.21.15.64:6790/0 conn(0x7fa6c013fd70 legacy=0x7fa6c01e2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:50.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.734+0000 7fa6a77fe640 1 -- 172.21.15.64:0/2633585968 >> v1:172.21.15.64:6789/0 conn(0x7fa6c013f990 legacy=0x7fa6c013d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:50.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.735+0000 7fa6a77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:10:50.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.735+0000 7fa6a77fe640 1 monclient: found mon.b 2024-03-25T18:10:50.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.735+0000 7fa6a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:50.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.735+0000 7fa6a77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:50.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.735+0000 7fa6a77fe640 1 -- 172.21.15.64:0/2633585968 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c011fad0 con 0x7fa6c00afa60 2024-03-25T18:10:50.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.735+0000 7fa6a77fe640 10 monclient: _finish_auth 0 2024-03-25T18:10:50.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.735+0000 7fa6a77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:10:50.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.735+0000 7fa6a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:50.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.735+0000 7fa6c5406640 5 monclient: authenticate success, global_id 4270 2024-03-25T18:10:50.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.735+0000 7fa6c5406640 10 monclient: _renew_subs 2024-03-25T18:10:50.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.735+0000 7fa6c5406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:50.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.735+0000 7fa6c5406640 1 -- 172.21.15.64:0/2633585968 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c01daf80 con 0x7fa6c00afa60 2024-03-25T18:10:50.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.735+0000 7fa6c5406640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:50.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.735+0000 7fa6c5406640 10 monclient: _renew_subs 2024-03-25T18:10:50.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.735+0000 7fa6c5406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:50.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.735+0000 7fa6c5406640 1 -- 172.21.15.64:0/2633585968 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c0143000 con 0x7fa6c00afa60 2024-03-25T18:10:50.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.735+0000 7fa6c5406640 1 librados: init done 2024-03-25T18:10:50.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.736+0000 7fa6c5406640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:50.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.736+0000 7fa6c5406640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:50.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.736+0000 7fa6c5406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:50.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:50.736+0000 7fa6c5406640 1 -- 172.21.15.64:0/2633585968 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c006bac0 con 0x7fa6c00afa60 2024-03-25T18:10:51.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.003+0000 7fa6a77fe640 1 -- 172.21.15.64:0/2633585968 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6b0001c70 con 0x7fa6c00afa60 2024-03-25T18:10:51.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.004+0000 7fa6a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:51.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.004+0000 7fa6bd7fa640 4 set_mon_vals no callback set 2024-03-25T18:10:51.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.004+0000 7fa6a77fe640 1 -- 172.21.15.64:0/2633585968 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fa6b0003a60 con 0x7fa6c00afa60 2024-03-25T18:10:51.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.004+0000 7fa6a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:51.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.004+0000 7fa6a77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:51.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.004+0000 7fa6a77fe640 10 monclient: dump: 2024-03-25T18:10:51.005 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:51.006 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:51.006 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:51.006 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:51.006 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:51.006 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:51.006 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:51.006 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:51.006 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:51.006 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:51.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.004+0000 7fa6a77fe640 1 -- 172.21.15.64:0/2633585968 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fa6b0003c80 con 0x7fa6c00afa60 2024-03-25T18:10:51.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.005+0000 7fa6be7fc640 10 monclient: get_auth_request con 0x7fa6ac06dd40 auth_method 0 2024-03-25T18:10:51.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.005+0000 7fa6a77fe640 10 client.4270.objecter ms_handle_connect 0x7fa6ac06dd40 2024-03-25T18:10:51.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.013+0000 7fa6a77fe640 1 -- 172.21.15.64:0/2633585968 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fa6b0054850 con 0x7fa6c00afa60 2024-03-25T18:10:51.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.013+0000 7fa6a77fe640 10 client.4270.objecter ms_dispatch 0x7fa6c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:51.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.013+0000 7fa6a77fe640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:51.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.013+0000 7fa6a77fe640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:51.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.013+0000 7fa6a77fe640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-25T18:10:51.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.014+0000 7fa6a77fe640 1 -- 172.21.15.64:0/2633585968 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6b0050360 con 0x7fa6c00afa60 2024-03-25T18:10:51.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.014+0000 7fa6a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:51.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.014+0000 7fa6a77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:51.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.271+0000 7fa6c5406640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:51.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.271+0000 7fa6c5406640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:51.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.271+0000 7fa6c5406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:51.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.271+0000 7fa6c5406640 1 -- 172.21.15.64:0/2633585968 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6c00627b0 con 0x7fa6c00afa60 2024-03-25T18:10:51.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.310+0000 7fa6a77fe640 1 -- 172.21.15.64:0/2633585968 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fa6b0052290 con 0x7fa6c00afa60 2024-03-25T18:10:51.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.310+0000 7fa6a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:51.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.310+0000 7fa6a77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:51.311 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:10:51.311 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:51.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.312+0000 7fa6c5406640 10 librados: watch_flush enter 2024-03-25T18:10:51.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.312+0000 7fa6c5406640 10 librados: watch_flush exit 2024-03-25T18:10:51.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.312+0000 7fa6c5406640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-25T18:10:51.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.312+0000 7fa6c5406640 10 client.4270.objecter successfully canceled tick 2024-03-25T18:10:51.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.312+0000 7fa6c5406640 1 -- 172.21.15.64:0/2633585968 >> v1:172.21.15.64:6800/2915543638 conn(0x7fa6ac06dd40 legacy=0x7fa6ac070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:51.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.312+0000 7fa6c5406640 10 monclient: shutdown 2024-03-25T18:10:51.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.312+0000 7fa6c5406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:51.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.312+0000 7fa6c5406640 1 -- 172.21.15.64:0/2633585968 >> v1:172.21.15.86:6789/0 conn(0x7fa6c00afa60 legacy=0x7fa6c01da870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:51.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.313+0000 7fa6c5406640 1 -- 172.21.15.64:0/2633585968 shutdown_connections 2024-03-25T18:10:51.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.313+0000 7fa6c5406640 1 -- 172.21.15.64:0/2633585968 >> 172.21.15.64:0/2633585968 conn(0x7fa6c00b4b80 msgr2=0x7fa6c00b4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:51.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.313+0000 7fa6c5406640 1 -- 172.21.15.64:0/2633585968 shutdown_connections 2024-03-25T18:10:51.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.313+0000 7fa6c5406640 1 -- 172.21.15.64:0/2633585968 wait complete. 2024-03-25T18:10:51.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:51.313+0000 7fa6c5406640 1 librados: shutdown 2024-03-25T18:10:51.325 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:51.425 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:01:34.538 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:51.429 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:01:34.542 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:51.468 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:01:34.581 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:51.471 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:01:34.584 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:51.475 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:01:34.554 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:51.483 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:01:34.562 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:51.484 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:01:34.597 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:51.526 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:01:34.605 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:51.536 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:01:34.649 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:58.131 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:58.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.237+0000 7fc82ef3a640 10 monclient: get_monmap_and_config 2024-03-25T18:10:58.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.237+0000 7fc82ef3a640 10 monclient: build_initial_monmap 2024-03-25T18:10:58.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.237+0000 7fc82ef3a640 10 monclient: monmap: 2024-03-25T18:10:58.238 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:10:58.238 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:58.238 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:10:58.237966+0000 2024-03-25T18:10:58.238 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:10:58.237966+0000 2024-03-25T18:10:58.238 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:58.238 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:58.238 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:10:58.238 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:10:58.238 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:10:58.238 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:58.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.238+0000 7fc82ef3a640 1 Processor -- start 2024-03-25T18:10:58.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.238+0000 7fc82ef3a640 1 -- start start 2024-03-25T18:10:58.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.238+0000 7fc82ef3a640 10 monclient: init 2024-03-25T18:10:58.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.238+0000 7fc82ef3a640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:58.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.238+0000 7fc82ef3a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:10:58.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.238+0000 7fc82ef3a640 10 monclient(hunting): picked mon.noname-a con 0x7fc82813fc60 addr v1:172.21.15.64:6789/0 2024-03-25T18:10:58.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.238+0000 7fc82ef3a640 10 monclient(hunting): picked mon.noname-b con 0x7fc828070570 addr v1:172.21.15.64:6790/0 2024-03-25T18:10:58.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.238+0000 7fc82ef3a640 10 monclient(hunting): picked mon.noname-c con 0x7fc8280727a0 addr v1:172.21.15.86:6789/0 2024-03-25T18:10:58.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.238+0000 7fc82ef3a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc82813c9f0 con 0x7fc82813fc60 2024-03-25T18:10:58.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.238+0000 7fc82ef3a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc82806fac0 con 0x7fc8280727a0 2024-03-25T18:10:58.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.239+0000 7fc82ef3a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8280715d0 con 0x7fc828070570 2024-03-25T18:10:58.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.239+0000 7fc82ef3a640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:58.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.239+0000 7fc82ef3a640 10 monclient(hunting): authenticate will time out at 7044.477506s 2024-03-25T18:10:58.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.244+0000 7fc82ccd6640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fc82813fc60 0x7fc82806f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:56668/0 (socket says 172.21.15.64:56668) 2024-03-25T18:10:58.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.244+0000 7fc82ccd6640 1 -- 172.21.15.64:0/920824775 learned_addr learned my addr 172.21.15.64:0/920824775 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:10:58.253 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.252+0000 7fc8277fe640 1 -- 172.21.15.64:0/920824775 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300761495 0 0) 0x7fc82813c9f0 con 0x7fc82813fc60 2024-03-25T18:10:58.253 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.252+0000 7fc8277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:58.253 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.252+0000 7fc8277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:58.253 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.252+0000 7fc8277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:58.253 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.252+0000 7fc8277fe640 10 monclient(hunting): my global_id is 4284 2024-03-25T18:10:58.253 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.252+0000 7fc8277fe640 1 -- 172.21.15.64:0/920824775 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc818003520 con 0x7fc82813fc60 2024-03-25T18:10:58.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.263+0000 7fc8277fe640 1 -- 172.21.15.64:0/920824775 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517323002 0 0) 0x7fc82806fac0 con 0x7fc8280727a0 2024-03-25T18:10:58.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.263+0000 7fc8277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:58.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.263+0000 7fc8277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:58.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.263+0000 7fc8277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:58.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.263+0000 7fc8277fe640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:10:58.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.263+0000 7fc8277fe640 1 -- 172.21.15.64:0/920824775 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc82813c9f0 con 0x7fc8280727a0 2024-03-25T18:10:58.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.267+0000 7fc8277fe640 1 -- 172.21.15.64:0/920824775 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 378839421 0 0) 0x7fc818003520 con 0x7fc82813fc60 2024-03-25T18:10:58.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.267+0000 7fc8277fe640 1 -- 172.21.15.64:0/920824775 >> v1:172.21.15.64:6790/0 conn(0x7fc828070570 legacy=0x7fc828072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:58.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.267+0000 7fc8277fe640 1 -- 172.21.15.64:0/920824775 >> v1:172.21.15.86:6789/0 conn(0x7fc8280727a0 legacy=0x7fc82814a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:58.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.267+0000 7fc8277fe640 10 monclient: _finish_hunting 0 2024-03-25T18:10:58.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.267+0000 7fc8277fe640 1 monclient: found mon.noname-a 2024-03-25T18:10:58.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.267+0000 7fc8277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:58.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.267+0000 7fc8277fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:10:58.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.267+0000 7fc8277fe640 1 -- 172.21.15.64:0/920824775 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8280718b0 con 0x7fc82813fc60 2024-03-25T18:10:58.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.267+0000 7fc8277fe640 10 monclient: _finish_auth 0 2024-03-25T18:10:58.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.267+0000 7fc8277fe640 10 monclient: _check_auth_tickets 2024-03-25T18:10:58.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.267+0000 7fc8277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:58.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.267+0000 7fc8277fe640 1 -- 172.21.15.64:0/920824775 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fc81c001690 con 0x7fc82813fc60 2024-03-25T18:10:58.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.267+0000 7fc8277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:58.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.267+0000 7fc8277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:58.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.267+0000 7fc8277fe640 10 monclient: dump: 2024-03-25T18:10:58.269 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:58.269 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:58.269 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:58.269 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:58.269 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:58.269 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:58.269 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:58.269 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:58.269 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:58.269 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:58.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.267+0000 7fc82ef3a640 5 monclient: authenticate success, global_id 4284 2024-03-25T18:10:58.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.267+0000 7fc82ef3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:58.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.280+0000 7fc8277fe640 1 -- 172.21.15.64:0/920824775 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc81c0017f0 con 0x7fc82813fc60 2024-03-25T18:10:58.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.280+0000 7fc8277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:58.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.280+0000 7fc8277fe640 1 -- 172.21.15.64:0/920824775 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fc81c001c00 con 0x7fc82813fc60 2024-03-25T18:10:58.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.280+0000 7fc8277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:58.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.280+0000 7fc8277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:58.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.280+0000 7fc8277fe640 10 monclient: dump: 2024-03-25T18:10:58.281 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:58.281 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:58.281 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:58.281 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:58.281 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:58.281 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:58.281 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:58.281 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:58.281 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:58.281 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:58.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.280+0000 7fc82ef3a640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:58.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.280+0000 7fc82ef3a640 4 set_mon_vals no callback set 2024-03-25T18:10:58.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.281+0000 7fc82ef3a640 10 monclient: shutdown 2024-03-25T18:10:58.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.281+0000 7fc82ef3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:58.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.281+0000 7fc82ef3a640 1 -- 172.21.15.64:0/920824775 >> v1:172.21.15.64:6789/0 conn(0x7fc82813fc60 legacy=0x7fc82806f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:58.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.281+0000 7fc82ef3a640 1 -- 172.21.15.64:0/920824775 shutdown_connections 2024-03-25T18:10:58.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.281+0000 7fc82ef3a640 1 -- 172.21.15.64:0/920824775 >> 172.21.15.64:0/920824775 conn(0x7fc8280b4a80 msgr2=0x7fc8280b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:58.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.281+0000 7fc82ef3a640 1 -- 172.21.15.64:0/920824775 shutdown_connections 2024-03-25T18:10:58.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.281+0000 7fc82ef3a640 1 -- 172.21.15.64:0/920824775 wait complete. 2024-03-25T18:10:58.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.281+0000 7fc82ef3a640 10 monclient: build_initial_monmap 2024-03-25T18:10:58.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.282+0000 7fc82ef3a640 10 monclient: monmap: 2024-03-25T18:10:58.283 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:10:58.283 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:58.284 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:10:58.284 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:10:58.284 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:58.284 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:10:58.284 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:10:58.284 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:10:58.284 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:10:58.284 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:58.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.282+0000 7fc82ef3a640 1 librados: starting msgr at 2024-03-25T18:10:58.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.282+0000 7fc82ef3a640 1 librados: starting objecter 2024-03-25T18:10:58.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.282+0000 7fc82ef3a640 1 Processor -- start 2024-03-25T18:10:58.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.282+0000 7fc82ef3a640 1 -- start start 2024-03-25T18:10:58.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.282+0000 7fc82ef3a640 1 librados: setting wanted keys 2024-03-25T18:10:58.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.282+0000 7fc82ef3a640 1 librados: calling monclient init 2024-03-25T18:10:58.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.282+0000 7fc82ef3a640 10 monclient: init 2024-03-25T18:10:58.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.283+0000 7fc82ef3a640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:58.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.283+0000 7fc82ef3a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:10:58.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.283+0000 7fc82ef3a640 10 monclient(hunting): picked mon.noname-a con 0x7fc828070570 addr v1:172.21.15.64:6789/0 2024-03-25T18:10:58.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.283+0000 7fc82ef3a640 10 monclient(hunting): picked mon.noname-b con 0x7fc8280727a0 addr v1:172.21.15.86:6789/0 2024-03-25T18:10:58.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.283+0000 7fc82ef3a640 10 monclient(hunting): picked mon.noname-c con 0x7fc82813fc60 addr v1:172.21.15.64:6790/0 2024-03-25T18:10:58.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.283+0000 7fc82ef3a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8281335e0 con 0x7fc828070570 2024-03-25T18:10:58.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.283+0000 7fc82ef3a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8281412d0 con 0x7fc8280727a0 2024-03-25T18:10:58.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.283+0000 7fc82ef3a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8281415b0 con 0x7fc82813fc60 2024-03-25T18:10:58.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.283+0000 7fc82ef3a640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:58.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.283+0000 7fc82ef3a640 10 monclient(hunting): authenticate will time out at 7044.521932s 2024-03-25T18:10:58.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.292+0000 7fc827fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc8280727a0 0x7fc8281327c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:39108/0 (socket says 172.21.15.64:39108) 2024-03-25T18:10:58.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.292+0000 7fc827fff640 1 -- 172.21.15.64:0/569968027 learned_addr learned my addr 172.21.15.64:0/569968027 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:10:58.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.294+0000 7fc8257fa640 10 client.?.objecter ms_handle_connect 0x7fc8280727a0 2024-03-25T18:10:58.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.294+0000 7fc8257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:58.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.306+0000 7fc8257fa640 10 client.?.objecter ms_handle_connect 0x7fc828070570 2024-03-25T18:10:58.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.306+0000 7fc8257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:58.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.308+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905116749 0 0) 0x7fc8281412d0 con 0x7fc8280727a0 2024-03-25T18:10:58.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.308+0000 7fc8257fa640 10 monclient(hunting): _negotiate 2024-03-25T18:10:58.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.308+0000 7fc8257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:58.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.308+0000 7fc8257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:58.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.308+0000 7fc8257fa640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:10:58.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.308+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc80c004eb0 con 0x7fc8280727a0 2024-03-25T18:10:58.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.315+0000 7fc8257fa640 10 client.?.objecter ms_handle_connect 0x7fc82813fc60 2024-03-25T18:10:58.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.315+0000 7fc8257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:58.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.319+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070826886 0 0) 0x7fc8281335e0 con 0x7fc828070570 2024-03-25T18:10:58.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.319+0000 7fc8257fa640 10 monclient(hunting): _negotiate 2024-03-25T18:10:58.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.319+0000 7fc8257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:58.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.319+0000 7fc8257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:58.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.319+0000 7fc8257fa640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:10:58.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.319+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8281412d0 con 0x7fc828070570 2024-03-25T18:10:58.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.328+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 464725633 0 0) 0x7fc80c004eb0 con 0x7fc8280727a0 2024-03-25T18:10:58.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.328+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8281335e0 con 0x7fc8280727a0 2024-03-25T18:10:58.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.328+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fc814001a30 con 0x7fc8280727a0 2024-03-25T18:10:58.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.328+0000 7fc8257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:58.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.328+0000 7fc8257fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:58.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.328+0000 7fc8257fa640 10 monclient(hunting): dump: 2024-03-25T18:10:58.329 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:58.329 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:58.329 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:58.329 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:58.329 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:58.329 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:58.330 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:58.330 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:58.330 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:58.330 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:58.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.328+0000 7fc8257fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:58.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.337+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2428235117 0 0) 0x7fc8281412d0 con 0x7fc828070570 2024-03-25T18:10:58.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.337+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc80c004eb0 con 0x7fc828070570 2024-03-25T18:10:58.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.337+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fc81c001b00 con 0x7fc828070570 2024-03-25T18:10:58.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.337+0000 7fc8257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:58.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.337+0000 7fc8257fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:58.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.337+0000 7fc8257fa640 10 monclient(hunting): dump: 2024-03-25T18:10:58.339 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:58.339 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:58.339 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:58.339 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:58.339 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:58.339 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:58.339 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:58.339 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:58.339 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:58.339 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:58.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2303550092 0 0) 0x7fc8281335e0 con 0x7fc8280727a0 2024-03-25T18:10:58.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 >> v1:172.21.15.64:6790/0 conn(0x7fc82813fc60 legacy=0x7fc828132ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:58.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 >> v1:172.21.15.64:6789/0 conn(0x7fc828070570 legacy=0x7fc8281d6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:58.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc8257fa640 10 monclient: _finish_hunting 0 2024-03-25T18:10:58.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc8257fa640 1 monclient: found mon.b 2024-03-25T18:10:58.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc8257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:58.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc8257fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:58.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc828140350 con 0x7fc8280727a0 2024-03-25T18:10:58.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc8257fa640 10 monclient: _finish_auth 0 2024-03-25T18:10:58.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc8257fa640 10 monclient: _check_auth_tickets 2024-03-25T18:10:58.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc8257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:58.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc82ef3a640 5 monclient: authenticate success, global_id 4276 2024-03-25T18:10:58.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc82ef3a640 10 monclient: _renew_subs 2024-03-25T18:10:58.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc82ef3a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:58.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc82ef3a640 1 -- 172.21.15.64:0/569968027 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8281cb030 con 0x7fc8280727a0 2024-03-25T18:10:58.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc82ef3a640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:58.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc82ef3a640 10 monclient: _renew_subs 2024-03-25T18:10:58.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc82ef3a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:58.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc82ef3a640 1 -- 172.21.15.64:0/569968027 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc828133a40 con 0x7fc8280727a0 2024-03-25T18:10:58.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.344+0000 7fc82ef3a640 1 librados: init done 2024-03-25T18:10:58.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.345+0000 7fc82ef3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:58.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.345+0000 7fc82ef3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:58.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.345+0000 7fc82ef3a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:58.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.345+0000 7fc82ef3a640 1 -- 172.21.15.64:0/569968027 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8280627b0 con 0x7fc8280727a0 2024-03-25T18:10:58.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.627+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc814001c70 con 0x7fc8280727a0 2024-03-25T18:10:58.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.627+0000 7fc8257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:58.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.627+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fc814003a60 con 0x7fc8280727a0 2024-03-25T18:10:58.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.627+0000 7fc8257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:58.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.627+0000 7fc8257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:58.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.627+0000 7fc8257fa640 10 monclient: dump: 2024-03-25T18:10:58.630 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:10:58.630 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:10:58.630 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:58.630 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:10:58.630 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:58.630 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:10:58.630 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:10:58.630 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:10:58.631 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:10:58.631 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:10:58.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.627+0000 7fc826ffd640 4 set_mon_vals no callback set 2024-03-25T18:10:58.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.628+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fc814003c80 con 0x7fc8280727a0 2024-03-25T18:10:58.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.628+0000 7fc82ccd6640 10 monclient: get_auth_request con 0x7fc80c06de90 auth_method 0 2024-03-25T18:10:58.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.629+0000 7fc8257fa640 10 client.4276.objecter ms_handle_connect 0x7fc80c06de90 2024-03-25T18:10:58.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.634+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fc814054850 con 0x7fc8280727a0 2024-03-25T18:10:58.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.634+0000 7fc8257fa640 10 client.4276.objecter ms_dispatch 0x7fc828000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:58.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.634+0000 7fc8257fa640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:58.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.634+0000 7fc8257fa640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:58.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.635+0000 7fc8257fa640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-25T18:10:58.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.635+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc814050360 con 0x7fc8280727a0 2024-03-25T18:10:58.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.635+0000 7fc8257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:58.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.635+0000 7fc8257fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:58.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.887+0000 7fc82ef3a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:58.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.887+0000 7fc82ef3a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:58.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.887+0000 7fc82ef3a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:10:58.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.887+0000 7fc82ef3a640 1 -- 172.21.15.64:0/569968027 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc82806b4d0 con 0x7fc8280727a0 2024-03-25T18:10:58.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.926+0000 7fc8257fa640 1 -- 172.21.15.64:0/569968027 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fc814052290 con 0x7fc8280727a0 2024-03-25T18:10:58.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.926+0000 7fc8257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:58.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.926+0000 7fc8257fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:58.928 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:10:58.929 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:58.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.928+0000 7fc82ef3a640 10 librados: watch_flush enter 2024-03-25T18:10:58.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.929+0000 7fc82ef3a640 10 librados: watch_flush exit 2024-03-25T18:10:58.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.929+0000 7fc82ef3a640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-25T18:10:58.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.929+0000 7fc82ef3a640 10 client.4276.objecter successfully canceled tick 2024-03-25T18:10:58.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.929+0000 7fc82ef3a640 1 -- 172.21.15.64:0/569968027 >> v1:172.21.15.64:6800/2915543638 conn(0x7fc80c06de90 legacy=0x7fc80c070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:58.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.929+0000 7fc82ef3a640 10 monclient: shutdown 2024-03-25T18:10:58.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.929+0000 7fc82ef3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:58.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.929+0000 7fc82ef3a640 1 -- 172.21.15.64:0/569968027 >> v1:172.21.15.86:6789/0 conn(0x7fc8280727a0 legacy=0x7fc8281327c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:58.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.929+0000 7fc82ef3a640 1 -- 172.21.15.64:0/569968027 shutdown_connections 2024-03-25T18:10:58.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.929+0000 7fc82ef3a640 1 -- 172.21.15.64:0/569968027 >> 172.21.15.64:0/569968027 conn(0x7fc8280b4a80 msgr2=0x7fc828074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:58.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.929+0000 7fc82ef3a640 1 -- 172.21.15.64:0/569968027 shutdown_connections 2024-03-25T18:10:58.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.929+0000 7fc82ef3a640 1 -- 172.21.15.64:0/569968027 wait complete. 2024-03-25T18:10:58.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:10:58.929+0000 7fc82ef3a640 1 librados: shutdown 2024-03-25T18:10:58.943 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:05.749 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:05.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.866+0000 7fe372da3640 10 monclient: get_monmap_and_config 2024-03-25T18:11:05.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.866+0000 7fe372da3640 10 monclient: build_initial_monmap 2024-03-25T18:11:05.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.867+0000 7fe372da3640 10 monclient: monmap: 2024-03-25T18:11:05.869 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:11:05.869 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:05.869 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:11:05.868867+0000 2024-03-25T18:11:05.869 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:11:05.868867+0000 2024-03-25T18:11:05.869 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:05.869 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:05.869 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:11:05.869 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:11:05.869 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:11:05.869 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:05.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.868+0000 7fe372da3640 1 Processor -- start 2024-03-25T18:11:05.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.868+0000 7fe372da3640 1 -- start start 2024-03-25T18:11:05.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.868+0000 7fe372da3640 10 monclient: init 2024-03-25T18:11:05.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.868+0000 7fe372da3640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:05.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.868+0000 7fe372da3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:11:05.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.868+0000 7fe372da3640 10 monclient(hunting): picked mon.noname-b con 0x7fe36c0a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:11:05.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.868+0000 7fe372da3640 10 monclient(hunting): picked mon.noname-a con 0x7fe36c13f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:11:05.870 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.868+0000 7fe372da3640 10 monclient(hunting): picked mon.noname-c con 0x7fe36c1430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:11:05.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.868+0000 7fe372da3640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36c142520 con 0x7fe36c13f710 2024-03-25T18:11:05.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.868+0000 7fe372da3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36c0b28a0 con 0x7fe36c1430c0 2024-03-25T18:11:05.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.868+0000 7fe372da3640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36c0b2a30 con 0x7fe36c0a98f0 2024-03-25T18:11:05.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.868+0000 7fe372da3640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:05.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.868+0000 7fe372da3640 10 monclient(hunting): authenticate will time out at 7052.108388s 2024-03-25T18:11:05.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.878+0000 7fe371340640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe36c1430c0 0x7fe36c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:43654/0 (socket says 172.21.15.64:43654) 2024-03-25T18:11:05.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.878+0000 7fe371340640 1 -- 172.21.15.64:0/1197431214 learned_addr learned my addr 172.21.15.64:0/1197431214 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:11:05.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.894+0000 7fe36b7fe640 1 -- 172.21.15.64:0/1197431214 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269144668 0 0) 0x7fe36c0b2a30 con 0x7fe36c0a98f0 2024-03-25T18:11:05.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.894+0000 7fe36b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:05.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.894+0000 7fe36b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:05.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.894+0000 7fe36b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:05.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.894+0000 7fe36b7fe640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:11:05.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.894+0000 7fe36b7fe640 1 -- 172.21.15.64:0/1197431214 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe35c003520 con 0x7fe36c0a98f0 2024-03-25T18:11:05.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.897+0000 7fe36b7fe640 1 -- 172.21.15.64:0/1197431214 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719804031 0 0) 0x7fe36c0b28a0 con 0x7fe36c1430c0 2024-03-25T18:11:05.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.897+0000 7fe36b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:05.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.897+0000 7fe36b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:05.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.897+0000 7fe36b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:05.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.897+0000 7fe36b7fe640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:11:05.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.897+0000 7fe36b7fe640 1 -- 172.21.15.64:0/1197431214 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe36c0b2a30 con 0x7fe36c1430c0 2024-03-25T18:11:05.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.907+0000 7fe36b7fe640 1 -- 172.21.15.64:0/1197431214 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2018030811 0 0) 0x7fe35c003520 con 0x7fe36c0a98f0 2024-03-25T18:11:05.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.907+0000 7fe36b7fe640 1 -- 172.21.15.64:0/1197431214 >> v1:172.21.15.86:6789/0 conn(0x7fe36c1430c0 legacy=0x7fe36c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:05.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.907+0000 7fe36b7fe640 1 -- 172.21.15.64:0/1197431214 >> v1:172.21.15.64:6789/0 conn(0x7fe36c13f710 legacy=0x7fe36c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:11:05.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.907+0000 7fe36b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:11:05.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.907+0000 7fe36b7fe640 1 monclient: found mon.noname-b 2024-03-25T18:11:05.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.907+0000 7fe36b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:05.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.907+0000 7fe36b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:11:05.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.907+0000 7fe36b7fe640 1 -- 172.21.15.64:0/1197431214 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c0b44c0 con 0x7fe36c0a98f0 2024-03-25T18:11:05.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.907+0000 7fe36b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:11:05.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.907+0000 7fe36b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:11:05.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.907+0000 7fe36b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:05.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.907+0000 7fe36b7fe640 1 -- 172.21.15.64:0/1197431214 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe360001690 con 0x7fe36c0a98f0 2024-03-25T18:11:05.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.907+0000 7fe36b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:05.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.907+0000 7fe36b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:05.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.907+0000 7fe36b7fe640 10 monclient: dump: 2024-03-25T18:11:05.910 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:05.910 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:05.910 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:05.910 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:05.910 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:05.910 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:05.911 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:05.911 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:05.911 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:05.911 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:05.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.907+0000 7fe372da3640 5 monclient: authenticate success, global_id 4289 2024-03-25T18:11:05.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.907+0000 7fe372da3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:05.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.925+0000 7fe36b7fe640 1 -- 172.21.15.64:0/1197431214 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3600017f0 con 0x7fe36c0a98f0 2024-03-25T18:11:05.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.925+0000 7fe36b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:05.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.925+0000 7fe36b7fe640 1 -- 172.21.15.64:0/1197431214 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe360001c00 con 0x7fe36c0a98f0 2024-03-25T18:11:05.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.925+0000 7fe372da3640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:05.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.925+0000 7fe372da3640 4 set_mon_vals no callback set 2024-03-25T18:11:05.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.925+0000 7fe36b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:05.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.925+0000 7fe36b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:05.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.925+0000 7fe36b7fe640 10 monclient: dump: 2024-03-25T18:11:05.927 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:05.927 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:05.928 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:05.928 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:05.928 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:05.928 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:05.928 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:05.928 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:05.928 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:05.928 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:05.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.926+0000 7fe372da3640 10 monclient: shutdown 2024-03-25T18:11:05.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.926+0000 7fe372da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:05.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.926+0000 7fe372da3640 1 -- 172.21.15.64:0/1197431214 >> v1:172.21.15.64:6790/0 conn(0x7fe36c0a98f0 legacy=0x7fe36c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:05.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.926+0000 7fe372da3640 1 -- 172.21.15.64:0/1197431214 shutdown_connections 2024-03-25T18:11:05.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.926+0000 7fe372da3640 1 -- 172.21.15.64:0/1197431214 >> 172.21.15.64:0/1197431214 conn(0x7fe36c0b7ea0 msgr2=0x7fe36c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:05.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.926+0000 7fe372da3640 1 -- 172.21.15.64:0/1197431214 shutdown_connections 2024-03-25T18:11:05.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.926+0000 7fe372da3640 1 -- 172.21.15.64:0/1197431214 wait complete. 2024-03-25T18:11:05.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.927+0000 7fe372da3640 10 monclient: build_initial_monmap 2024-03-25T18:11:05.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.927+0000 7fe372da3640 10 monclient: monmap: 2024-03-25T18:11:05.931 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:11:05.931 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:05.931 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:11:05.931 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:11:05.931 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:05.931 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:11:05.931 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:11:05.931 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:11:05.931 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:11:05.931 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:05.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.927+0000 7fe372da3640 1 librados: starting msgr at 2024-03-25T18:11:05.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.927+0000 7fe372da3640 1 librados: starting objecter 2024-03-25T18:11:05.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.927+0000 7fe372da3640 1 Processor -- start 2024-03-25T18:11:05.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.927+0000 7fe372da3640 1 -- start start 2024-03-25T18:11:05.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.927+0000 7fe372da3640 1 librados: setting wanted keys 2024-03-25T18:11:05.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.927+0000 7fe372da3640 1 librados: calling monclient init 2024-03-25T18:11:05.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.927+0000 7fe372da3640 10 monclient: init 2024-03-25T18:11:05.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.928+0000 7fe372da3640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:05.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.928+0000 7fe372da3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:11:05.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.928+0000 7fe372da3640 10 monclient(hunting): picked mon.noname-c con 0x7fe36c0a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:11:05.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.928+0000 7fe372da3640 10 monclient(hunting): picked mon.noname-b con 0x7fe36c13f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:11:05.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.928+0000 7fe372da3640 10 monclient(hunting): picked mon.noname-a con 0x7fe36c1430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:11:05.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.928+0000 7fe372da3640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36c1d8400 con 0x7fe36c1430c0 2024-03-25T18:11:05.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.928+0000 7fe372da3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36c1d7df0 con 0x7fe36c13f710 2024-03-25T18:11:05.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.928+0000 7fe372da3640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36c1d7f80 con 0x7fe36c0a98f0 2024-03-25T18:11:05.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.928+0000 7fe372da3640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:05.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.928+0000 7fe372da3640 10 monclient(hunting): authenticate will time out at 7052.168153s 2024-03-25T18:11:05.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.935+0000 7fe36bfff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe36c13f710 0x7fe36c132f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:43692/0 (socket says 172.21.15.64:43692) 2024-03-25T18:11:05.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.935+0000 7fe36bfff640 1 -- 172.21.15.64:0/3612894874 learned_addr learned my addr 172.21.15.64:0/3612894874 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:11:05.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.935+0000 7fe3697fa640 10 client.?.objecter ms_handle_connect 0x7fe36c13f710 2024-03-25T18:11:05.937 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.935+0000 7fe3697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:05.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.943+0000 7fe3697fa640 10 client.?.objecter ms_handle_connect 0x7fe36c1430c0 2024-03-25T18:11:05.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:05.943+0000 7fe3697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:06.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.031+0000 7fe3697fa640 10 client.?.objecter ms_handle_connect 0x7fe36c0a98f0 2024-03-25T18:11:06.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.031+0000 7fe3697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:06.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.040+0000 7fe3697fa640 1 -- 172.21.15.64:0/3612894874 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302065221 0 0) 0x7fe36c1d7f80 con 0x7fe36c0a98f0 2024-03-25T18:11:06.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.040+0000 7fe3697fa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:06.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.040+0000 7fe3697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:06.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.040+0000 7fe3697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:06.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.040+0000 7fe3697fa640 10 monclient(hunting): my global_id is 4292 2024-03-25T18:11:06.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.040+0000 7fe3697fa640 1 -- 172.21.15.64:0/3612894874 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe350004eb0 con 0x7fe36c0a98f0 2024-03-25T18:11:06.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.060+0000 7fe3697fa640 1 -- 172.21.15.64:0/3612894874 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2486676896 0 0) 0x7fe350004eb0 con 0x7fe36c0a98f0 2024-03-25T18:11:06.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.060+0000 7fe3697fa640 1 -- 172.21.15.64:0/3612894874 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe36c1d7f80 con 0x7fe36c0a98f0 2024-03-25T18:11:06.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.061+0000 7fe3697fa640 1 -- 172.21.15.64:0/3612894874 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe360001bc0 con 0x7fe36c0a98f0 2024-03-25T18:11:06.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.061+0000 7fe3697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:06.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.061+0000 7fe3697fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:06.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.061+0000 7fe3697fa640 10 monclient(hunting): dump: 2024-03-25T18:11:06.062 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:06.062 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:06.062 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:06.062 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:06.062 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:06.062 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:06.062 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:06.062 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:06.062 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:06.062 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:06.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.061+0000 7fe3697fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:06.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.074+0000 7fe3697fa640 1 -- 172.21.15.64:0/3612894874 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3050685888 0 0) 0x7fe36c1d7f80 con 0x7fe36c0a98f0 2024-03-25T18:11:06.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.074+0000 7fe3697fa640 1 -- 172.21.15.64:0/3612894874 >> v1:172.21.15.86:6789/0 conn(0x7fe36c13f710 legacy=0x7fe36c132f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:06.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.074+0000 7fe3697fa640 1 -- 172.21.15.64:0/3612894874 >> v1:172.21.15.64:6789/0 conn(0x7fe36c1430c0 legacy=0x7fe36c135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:06.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.074+0000 7fe3697fa640 10 monclient: _finish_hunting 0 2024-03-25T18:11:06.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.074+0000 7fe3697fa640 1 monclient: found mon.c 2024-03-25T18:11:06.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.074+0000 7fe3697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:06.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.074+0000 7fe3697fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:11:06.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.074+0000 7fe3697fa640 1 -- 172.21.15.64:0/3612894874 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c1d8110 con 0x7fe36c0a98f0 2024-03-25T18:11:06.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.074+0000 7fe3697fa640 10 monclient: _finish_auth 0 2024-03-25T18:11:06.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.074+0000 7fe3697fa640 10 monclient: _check_auth_tickets 2024-03-25T18:11:06.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.074+0000 7fe3697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:06.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.074+0000 7fe372da3640 5 monclient: authenticate success, global_id 4292 2024-03-25T18:11:06.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.074+0000 7fe372da3640 10 monclient: _renew_subs 2024-03-25T18:11:06.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.074+0000 7fe372da3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:11:06.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.074+0000 7fe372da3640 1 -- 172.21.15.64:0/3612894874 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36c0b7300 con 0x7fe36c0a98f0 2024-03-25T18:11:06.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.075+0000 7fe372da3640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:06.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.075+0000 7fe372da3640 10 monclient: _renew_subs 2024-03-25T18:11:06.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.075+0000 7fe372da3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:11:06.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.075+0000 7fe372da3640 1 -- 172.21.15.64:0/3612894874 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36c1336a0 con 0x7fe36c0a98f0 2024-03-25T18:11:06.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.075+0000 7fe372da3640 1 librados: init done 2024-03-25T18:11:06.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.075+0000 7fe372da3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:06.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.075+0000 7fe372da3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:06.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.075+0000 7fe372da3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:11:06.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.075+0000 7fe372da3640 1 -- 172.21.15.64:0/3612894874 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe36c0b3ef0 con 0x7fe36c0a98f0 2024-03-25T18:11:06.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.348+0000 7fe3697fa640 1 -- 172.21.15.64:0/3612894874 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3600038a0 con 0x7fe36c0a98f0 2024-03-25T18:11:06.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.348+0000 7fe3697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:06.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.348+0000 7fe3697fa640 1 -- 172.21.15.64:0/3612894874 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe360003bb0 con 0x7fe36c0a98f0 2024-03-25T18:11:06.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.348+0000 7fe3697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:06.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.348+0000 7fe3697fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:06.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.348+0000 7fe3697fa640 10 monclient: dump: 2024-03-25T18:11:06.350 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:06.350 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:06.350 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:06.350 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:06.350 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:06.350 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:06.350 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:06.350 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:06.350 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:06.350 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:06.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.348+0000 7fe36affd640 4 set_mon_vals no callback set 2024-03-25T18:11:06.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.349+0000 7fe3697fa640 1 -- 172.21.15.64:0/3612894874 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fe360019b80 con 0x7fe36c0a98f0 2024-03-25T18:11:06.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.349+0000 7fe36bfff640 10 monclient: get_auth_request con 0x7fe35006dc90 auth_method 0 2024-03-25T18:11:06.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.350+0000 7fe3697fa640 10 client.4292.objecter ms_handle_connect 0x7fe35006dc90 2024-03-25T18:11:06.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.363+0000 7fe3697fa640 1 -- 172.21.15.64:0/3612894874 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fe360053740 con 0x7fe36c0a98f0 2024-03-25T18:11:06.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.363+0000 7fe3697fa640 10 client.4292.objecter ms_dispatch 0x7fe36c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:06.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.363+0000 7fe3697fa640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:06.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.363+0000 7fe3697fa640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:06.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.363+0000 7fe3697fa640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-25T18:11:06.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.364+0000 7fe3697fa640 1 -- 172.21.15.64:0/3612894874 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe36004f080 con 0x7fe36c0a98f0 2024-03-25T18:11:06.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.364+0000 7fe3697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:06.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.364+0000 7fe3697fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:06.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.594+0000 7fe372da3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:06.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.594+0000 7fe372da3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:06.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.595+0000 7fe372da3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:11:06.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.595+0000 7fe372da3640 1 -- 172.21.15.64:0/3612894874 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe36c0627b0 con 0x7fe36c0a98f0 2024-03-25T18:11:06.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.616+0000 7fe3697fa640 1 -- 172.21.15.64:0/3612894874 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fe360051180 con 0x7fe36c0a98f0 2024-03-25T18:11:06.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.616+0000 7fe3697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:06.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.616+0000 7fe3697fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:06.617 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:11:06.617 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:06.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.618+0000 7fe372da3640 10 librados: watch_flush enter 2024-03-25T18:11:06.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.618+0000 7fe372da3640 10 librados: watch_flush exit 2024-03-25T18:11:06.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.619+0000 7fe372da3640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-25T18:11:06.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.619+0000 7fe372da3640 10 client.4292.objecter successfully canceled tick 2024-03-25T18:11:06.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.619+0000 7fe372da3640 1 -- 172.21.15.64:0/3612894874 >> v1:172.21.15.64:6800/2915543638 conn(0x7fe35006dc90 legacy=0x7fe350070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:06.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.619+0000 7fe372da3640 10 monclient: shutdown 2024-03-25T18:11:06.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.619+0000 7fe372da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:06.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.619+0000 7fe372da3640 1 -- 172.21.15.64:0/3612894874 >> v1:172.21.15.64:6790/0 conn(0x7fe36c0a98f0 legacy=0x7fe36c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:06.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.619+0000 7fe372da3640 1 -- 172.21.15.64:0/3612894874 shutdown_connections 2024-03-25T18:11:06.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.619+0000 7fe372da3640 1 -- 172.21.15.64:0/3612894874 >> 172.21.15.64:0/3612894874 conn(0x7fe36c0b7ea0 msgr2=0x7fe36c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:06.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.619+0000 7fe372da3640 1 -- 172.21.15.64:0/3612894874 shutdown_connections 2024-03-25T18:11:06.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.619+0000 7fe372da3640 1 -- 172.21.15.64:0/3612894874 wait complete. 2024-03-25T18:11:06.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:06.619+0000 7fe372da3640 1 librados: shutdown 2024-03-25T18:11:06.631 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:11.494 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:01:54.607 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:11.499 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:01:54.612 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:11.521 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:01:54.634 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:11.551 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:01:54.630 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:11.554 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:01:54.667 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:11.557 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:01:54.636 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:11.562 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:01:54.675 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:11.585 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:01:54.664 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:11.605 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:01:54.718 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:13.437 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:13.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.558+0000 7f026604a640 10 monclient: get_monmap_and_config 2024-03-25T18:11:13.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.558+0000 7f026604a640 10 monclient: build_initial_monmap 2024-03-25T18:11:13.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.558+0000 7f026604a640 10 monclient: monmap: 2024-03-25T18:11:13.559 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:11:13.559 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:13.559 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:11:13.559453+0000 2024-03-25T18:11:13.559 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:11:13.559453+0000 2024-03-25T18:11:13.559 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:13.559 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:13.559 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:11:13.559 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:11:13.559 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:11:13.559 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:13.560 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.559+0000 7f026604a640 1 Processor -- start 2024-03-25T18:11:13.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.559+0000 7f026604a640 1 -- start start 2024-03-25T18:11:13.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.559+0000 7f026604a640 10 monclient: init 2024-03-25T18:11:13.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.559+0000 7f026604a640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:13.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.559+0000 7f026604a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:13.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.559+0000 7f026604a640 10 monclient(hunting): picked mon.noname-a con 0x7f02600a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:11:13.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.560+0000 7f026604a640 10 monclient(hunting): picked mon.noname-b con 0x7f026013f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:11:13.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.560+0000 7f026604a640 10 monclient(hunting): picked mon.noname-c con 0x7f02601430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:11:13.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.560+0000 7f026604a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0260142520 con 0x7f02600a98f0 2024-03-25T18:11:13.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.560+0000 7f026604a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02600b28a0 con 0x7f02601430c0 2024-03-25T18:11:13.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.560+0000 7f026604a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02600b2a30 con 0x7f026013f710 2024-03-25T18:11:13.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.560+0000 7f026604a640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:13.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.560+0000 7f026604a640 10 monclient(hunting): authenticate will time out at 7059.798532s 2024-03-25T18:11:13.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.571+0000 7f025ffff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f02601430c0 0x7f026014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:48544/0 (socket says 172.21.15.64:48544) 2024-03-25T18:11:13.571 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.571+0000 7f025ffff640 1 -- 172.21.15.64:0/920946822 learned_addr learned my addr 172.21.15.64:0/920946822 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:11:13.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.582+0000 7f025e7fc640 1 -- 172.21.15.64:0/920946822 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451868786 0 0) 0x7f02600b28a0 con 0x7f02601430c0 2024-03-25T18:11:13.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.583+0000 7f025e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:13.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.583+0000 7f025e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:13.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.583+0000 7f025e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:13.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.583+0000 7f025e7fc640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:11:13.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.583+0000 7f025e7fc640 1 -- 172.21.15.64:0/920946822 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0248003520 con 0x7f02601430c0 2024-03-25T18:11:13.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.591+0000 7f025e7fc640 1 -- 172.21.15.64:0/920946822 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032867418 0 0) 0x7f0260142520 con 0x7f02600a98f0 2024-03-25T18:11:13.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.591+0000 7f025e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:13.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.591+0000 7f025e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:13.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.591+0000 7f025e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:13.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.591+0000 7f025e7fc640 10 monclient(hunting): my global_id is 4293 2024-03-25T18:11:13.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.591+0000 7f025e7fc640 1 -- 172.21.15.64:0/920946822 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02600b28a0 con 0x7f02600a98f0 2024-03-25T18:11:13.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.594+0000 7f025e7fc640 1 -- 172.21.15.64:0/920946822 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117329844 0 0) 0x7f02600b2a30 con 0x7f026013f710 2024-03-25T18:11:13.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.594+0000 7f025e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:13.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.594+0000 7f025e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:13.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.594+0000 7f025e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:13.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.594+0000 7f025e7fc640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:11:13.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.594+0000 7f025e7fc640 1 -- 172.21.15.64:0/920946822 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0260142520 con 0x7f026013f710 2024-03-25T18:11:13.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.602+0000 7f025e7fc640 1 -- 172.21.15.64:0/920946822 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 177625807 0 0) 0x7f0248003520 con 0x7f02601430c0 2024-03-25T18:11:13.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.602+0000 7f025e7fc640 1 -- 172.21.15.64:0/920946822 >> v1:172.21.15.64:6790/0 conn(0x7f026013f710 legacy=0x7f02601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:13.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.602+0000 7f025e7fc640 1 -- 172.21.15.64:0/920946822 >> v1:172.21.15.64:6789/0 conn(0x7f02600a98f0 legacy=0x7f02600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:13.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.602+0000 7f025e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:11:13.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.602+0000 7f025e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:11:13.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.602+0000 7f025e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:13.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.602+0000 7f025e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:11:13.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.602+0000 7f025e7fc640 1 -- 172.21.15.64:0/920946822 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02600b44c0 con 0x7f02601430c0 2024-03-25T18:11:13.603 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.602+0000 7f025e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:11:13.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.602+0000 7f025e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:11:13.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.602+0000 7f025e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:13.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.602+0000 7f025e7fc640 1 -- 172.21.15.64:0/920946822 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0258001a10 con 0x7f02601430c0 2024-03-25T18:11:13.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.602+0000 7f025e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:13.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.602+0000 7f025e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:13.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.602+0000 7f025e7fc640 10 monclient: dump: 2024-03-25T18:11:13.604 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:13.604 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:13.604 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:13.604 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:13.604 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:13.604 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:13.604 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:13.604 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:13.604 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:13.605 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:13.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.602+0000 7f026604a640 5 monclient: authenticate success, global_id 4285 2024-03-25T18:11:13.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.602+0000 7f026604a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:13.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.611+0000 7f025e7fc640 1 -- 172.21.15.64:0/920946822 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0258001b70 con 0x7f02601430c0 2024-03-25T18:11:13.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.611+0000 7f025e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:13.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.611+0000 7f025e7fc640 1 -- 172.21.15.64:0/920946822 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0258001f80 con 0x7f02601430c0 2024-03-25T18:11:13.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.611+0000 7f025e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:13.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.611+0000 7f025e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:13.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.611+0000 7f025e7fc640 10 monclient: dump: 2024-03-25T18:11:13.612 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:13.612 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:13.612 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:13.612 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:13.612 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:13.612 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:13.612 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:13.612 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:13.612 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:13.612 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:13.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.611+0000 7f026604a640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:13.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.611+0000 7f026604a640 4 set_mon_vals no callback set 2024-03-25T18:11:13.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.612+0000 7f026604a640 10 monclient: shutdown 2024-03-25T18:11:13.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.612+0000 7f026604a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:13.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.612+0000 7f026604a640 1 -- 172.21.15.64:0/920946822 >> v1:172.21.15.86:6789/0 conn(0x7f02601430c0 legacy=0x7f026014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:13.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.612+0000 7f026604a640 1 -- 172.21.15.64:0/920946822 shutdown_connections 2024-03-25T18:11:13.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.612+0000 7f026604a640 1 -- 172.21.15.64:0/920946822 >> 172.21.15.64:0/920946822 conn(0x7f02600b7ea0 msgr2=0x7f02600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:13.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.612+0000 7f026604a640 1 -- 172.21.15.64:0/920946822 shutdown_connections 2024-03-25T18:11:13.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.612+0000 7f026604a640 1 -- 172.21.15.64:0/920946822 wait complete. 2024-03-25T18:11:13.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.613+0000 7f026604a640 10 monclient: build_initial_monmap 2024-03-25T18:11:13.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.613+0000 7f026604a640 10 monclient: monmap: 2024-03-25T18:11:13.616 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:11:13.616 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:13.616 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:11:13.616 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:11:13.616 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:13.616 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:11:13.616 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:11:13.616 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:11:13.616 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:11:13.616 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:13.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.613+0000 7f026604a640 1 librados: starting msgr at 2024-03-25T18:11:13.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.613+0000 7f026604a640 1 librados: starting objecter 2024-03-25T18:11:13.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.613+0000 7f026604a640 1 Processor -- start 2024-03-25T18:11:13.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.613+0000 7f026604a640 1 -- start start 2024-03-25T18:11:13.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.613+0000 7f026604a640 1 librados: setting wanted keys 2024-03-25T18:11:13.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.613+0000 7f026604a640 1 librados: calling monclient init 2024-03-25T18:11:13.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.613+0000 7f026604a640 10 monclient: init 2024-03-25T18:11:13.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.614+0000 7f026604a640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:13.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.614+0000 7f026604a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:13.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.614+0000 7f026604a640 10 monclient(hunting): picked mon.noname-a con 0x7f02600a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:11:13.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.614+0000 7f026604a640 10 monclient(hunting): picked mon.noname-c con 0x7f026013f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:11:13.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.614+0000 7f026604a640 10 monclient(hunting): picked mon.noname-b con 0x7f02601430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:11:13.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.614+0000 7f026604a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02601369d0 con 0x7f02600a98f0 2024-03-25T18:11:13.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.614+0000 7f026604a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0260133a60 con 0x7f02601430c0 2024-03-25T18:11:13.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.614+0000 7f026604a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02600ab9e0 con 0x7f026013f710 2024-03-25T18:11:13.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.614+0000 7f026604a640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:13.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.614+0000 7f026604a640 10 monclient(hunting): authenticate will time out at 7059.852782s 2024-03-25T18:11:13.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.631+0000 7f025ffff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f02601430c0 0x7f0260133350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:48570/0 (socket says 172.21.15.64:48570) 2024-03-25T18:11:13.632 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.631+0000 7f025ffff640 1 -- 172.21.15.64:0/824151191 learned_addr learned my addr 172.21.15.64:0/824151191 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:11:13.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.632+0000 7f0247fff640 10 client.?.objecter ms_handle_connect 0x7f02601430c0 2024-03-25T18:11:13.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.632+0000 7f0247fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:13.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.633+0000 7f0247fff640 10 client.?.objecter ms_handle_connect 0x7f026013f710 2024-03-25T18:11:13.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.633+0000 7f0247fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:13.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.642+0000 7f0247fff640 10 client.?.objecter ms_handle_connect 0x7f02600a98f0 2024-03-25T18:11:13.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.642+0000 7f0247fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:13.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.649+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666584884 0 0) 0x7f0260133a60 con 0x7f02601430c0 2024-03-25T18:11:13.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.649+0000 7f0247fff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:13.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.649+0000 7f0247fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:13.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.649+0000 7f0247fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:13.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.649+0000 7f0247fff640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:11:13.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.649+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f024c004eb0 con 0x7f02601430c0 2024-03-25T18:11:13.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.655+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911840195 0 0) 0x7f02601369d0 con 0x7f02600a98f0 2024-03-25T18:11:13.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.655+0000 7f0247fff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:13.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.655+0000 7f0247fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:13.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.655+0000 7f0247fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:13.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.655+0000 7f0247fff640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:11:13.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.655+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0260133a60 con 0x7f02600a98f0 2024-03-25T18:11:13.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.658+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114905721 0 0) 0x7f02600ab9e0 con 0x7f026013f710 2024-03-25T18:11:13.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.658+0000 7f0247fff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:13.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.658+0000 7f0247fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:13.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.658+0000 7f0247fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:13.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.658+0000 7f0247fff640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:11:13.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.658+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02601369d0 con 0x7f026013f710 2024-03-25T18:11:13.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.661+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2839867009 0 0) 0x7f024c004eb0 con 0x7f02601430c0 2024-03-25T18:11:13.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.661+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02600ab9e0 con 0x7f02601430c0 2024-03-25T18:11:13.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.661+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0258001ec0 con 0x7f02601430c0 2024-03-25T18:11:13.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.661+0000 7f0247fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:13.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.661+0000 7f0247fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:13.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.661+0000 7f0247fff640 10 monclient(hunting): dump: 2024-03-25T18:11:13.663 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:13.663 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:13.663 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:13.663 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:13.663 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:13.663 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:13.663 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:13.663 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:13.663 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:13.663 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:13.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.661+0000 7f0247fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:13.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.668+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 549637977 0 0) 0x7f0260133a60 con 0x7f02600a98f0 2024-03-25T18:11:13.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.668+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f024c004eb0 con 0x7f02600a98f0 2024-03-25T18:11:13.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.668+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0250001970 con 0x7f02600a98f0 2024-03-25T18:11:13.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.668+0000 7f0247fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:13.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.668+0000 7f0247fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:13.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.668+0000 7f0247fff640 10 monclient(hunting): dump: 2024-03-25T18:11:13.669 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:13.669 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:13.669 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:13.669 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:13.670 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:13.670 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:13.670 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:13.670 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:13.670 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:13.670 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:13.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3712484834 0 0) 0x7f024c004eb0 con 0x7f02600a98f0 2024-03-25T18:11:13.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 >> v1:172.21.15.64:6790/0 conn(0x7f026013f710 legacy=0x7f02601362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:13.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 >> v1:172.21.15.86:6789/0 conn(0x7f02601430c0 legacy=0x7f0260133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:13.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f0247fff640 10 monclient: _finish_hunting 0 2024-03-25T18:11:13.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f0247fff640 1 monclient: found mon.a 2024-03-25T18:11:13.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f0247fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:13.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f0247fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:11:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02600abcc0 con 0x7f02600a98f0 2024-03-25T18:11:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f0247fff640 10 monclient: _finish_auth 0 2024-03-25T18:11:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f0247fff640 10 monclient: _check_auth_tickets 2024-03-25T18:11:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f0247fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f026604a640 5 monclient: authenticate success, global_id 4296 2024-03-25T18:11:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f026604a640 10 monclient: _renew_subs 2024-03-25T18:11:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f026604a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:11:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f026604a640 1 -- 172.21.15.64:0/824151191 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02600abfa0 con 0x7f02600a98f0 2024-03-25T18:11:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f026604a640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f026604a640 10 monclient: _renew_subs 2024-03-25T18:11:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f026604a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:11:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f026604a640 1 -- 172.21.15.64:0/824151191 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0260137e40 con 0x7f02600a98f0 2024-03-25T18:11:13.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.676+0000 7f026604a640 1 librados: init done 2024-03-25T18:11:13.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.677+0000 7f026604a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:13.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.677+0000 7f026604a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:13.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.677+0000 7f026604a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:11:13.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.677+0000 7f026604a640 1 -- 172.21.15.64:0/824151191 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02600627b0 con 0x7f02600a98f0 2024-03-25T18:11:13.975 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.974+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0250003610 con 0x7f02600a98f0 2024-03-25T18:11:13.975 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.974+0000 7f0247fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:13.975 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.974+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0250003920 con 0x7f02600a98f0 2024-03-25T18:11:13.975 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.974+0000 7f0247fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:13.975 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.974+0000 7f0247fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:13.975 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.974+0000 7f0247fff640 10 monclient: dump: 2024-03-25T18:11:13.975 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:13.976 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:13.976 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:13.976 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:13.976 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:13.976 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:13.976 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:13.976 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:13.976 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:13.976 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:13.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.974+0000 7f025dffb640 4 set_mon_vals no callback set 2024-03-25T18:11:13.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.975+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f02500198f0 con 0x7f02600a98f0 2024-03-25T18:11:13.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.976+0000 7f025effd640 10 monclient: get_auth_request con 0x7f024c06df70 auth_method 0 2024-03-25T18:11:13.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.976+0000 7f0247fff640 10 client.4296.objecter ms_handle_connect 0x7f024c06df70 2024-03-25T18:11:13.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.981+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f02500535e0 con 0x7f02600a98f0 2024-03-25T18:11:13.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.981+0000 7f0247fff640 10 client.4296.objecter ms_dispatch 0x7f0260000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:13.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.981+0000 7f0247fff640 3 client.4296.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:13.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.981+0000 7f0247fff640 3 client.4296.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:13.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.981+0000 7f0247fff640 20 client.4296.objecter dump_active .. 0 homeless 2024-03-25T18:11:13.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.981+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f025004ef20 con 0x7f02600a98f0 2024-03-25T18:11:13.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.982+0000 7f0247fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:13.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:13.982+0000 7f0247fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:14.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.208+0000 7f026604a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:14.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.208+0000 7f026604a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:14.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.208+0000 7f026604a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:11:14.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.208+0000 7f026604a640 1 -- 172.21.15.64:0/824151191 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f026013dfd0 con 0x7f02600a98f0 2024-03-25T18:11:14.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.239+0000 7f0247fff640 1 -- 172.21.15.64:0/824151191 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f0250051020 con 0x7f02600a98f0 2024-03-25T18:11:14.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.239+0000 7f0247fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:14.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.239+0000 7f0247fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:14.240 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:11:14.240 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:14.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.241+0000 7f026604a640 10 librados: watch_flush enter 2024-03-25T18:11:14.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.241+0000 7f026604a640 10 librados: watch_flush exit 2024-03-25T18:11:14.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.241+0000 7f026604a640 20 client.4296.objecter shutdown clearing up homeless session... 2024-03-25T18:11:14.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.241+0000 7f026604a640 10 client.4296.objecter successfully canceled tick 2024-03-25T18:11:14.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.241+0000 7f026604a640 1 -- 172.21.15.64:0/824151191 >> v1:172.21.15.64:6800/2915543638 conn(0x7f024c06df70 legacy=0x7f024c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:14.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.241+0000 7f026604a640 10 monclient: shutdown 2024-03-25T18:11:14.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.241+0000 7f026604a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:14.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.241+0000 7f026604a640 1 -- 172.21.15.64:0/824151191 >> v1:172.21.15.64:6789/0 conn(0x7f02600a98f0 legacy=0x7f0260132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:14.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.241+0000 7f026604a640 1 -- 172.21.15.64:0/824151191 shutdown_connections 2024-03-25T18:11:14.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.241+0000 7f026604a640 1 -- 172.21.15.64:0/824151191 >> 172.21.15.64:0/824151191 conn(0x7f02600b7ea0 msgr2=0x7f0260074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:14.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.241+0000 7f026604a640 1 -- 172.21.15.64:0/824151191 shutdown_connections 2024-03-25T18:11:14.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.241+0000 7f026604a640 1 -- 172.21.15.64:0/824151191 wait complete. 2024-03-25T18:11:14.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:14.241+0000 7f026604a640 1 librados: shutdown 2024-03-25T18:11:14.254 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:21.060 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:21.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.172+0000 7feaadb47640 10 monclient: get_monmap_and_config 2024-03-25T18:11:21.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.172+0000 7feaadb47640 10 monclient: build_initial_monmap 2024-03-25T18:11:21.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.172+0000 7feaadb47640 10 monclient: monmap: 2024-03-25T18:11:21.172 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:11:21.172 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:21.172 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:11:21.172913+0000 2024-03-25T18:11:21.173 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:11:21.172913+0000 2024-03-25T18:11:21.173 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:21.173 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:21.173 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:11:21.173 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:11:21.173 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:11:21.173 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:21.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.172+0000 7feaadb47640 1 Processor -- start 2024-03-25T18:11:21.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.172+0000 7feaadb47640 1 -- start start 2024-03-25T18:11:21.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.172+0000 7feaadb47640 10 monclient: init 2024-03-25T18:11:21.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.173+0000 7feaadb47640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:21.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.173+0000 7feaadb47640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:21.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.173+0000 7feaadb47640 10 monclient(hunting): picked mon.noname-a con 0x7feaa80b4d60 addr v1:172.21.15.64:6789/0 2024-03-25T18:11:21.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.173+0000 7feaadb47640 10 monclient(hunting): picked mon.noname-c con 0x7feaa80b1730 addr v1:172.21.15.86:6789/0 2024-03-25T18:11:21.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.173+0000 7feaadb47640 10 monclient(hunting): picked mon.noname-b con 0x7feaa814be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:11:21.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.173+0000 7feaadb47640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa80af940 con 0x7feaa80b4d60 2024-03-25T18:11:21.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.173+0000 7feaadb47640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa80b0a70 con 0x7feaa80b1730 2024-03-25T18:11:21.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.173+0000 7feaadb47640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa80b0d50 con 0x7feaa814be90 2024-03-25T18:11:21.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.173+0000 7feaadb47640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:21.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.173+0000 7feaadb47640 10 monclient(hunting): authenticate will time out at 7067.412018s 2024-03-25T18:11:21.184 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.184+0000 7feaa7fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7feaa814be90 0x7feaa814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:42880/0 (socket says 172.21.15.64:42880) 2024-03-25T18:11:21.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.184+0000 7feaa7fff640 1 -- 172.21.15.64:0/2988354742 learned_addr learned my addr 172.21.15.64:0/2988354742 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:11:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.193+0000 7feaa67fc640 1 -- 172.21.15.64:0/2988354742 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44663667 0 0) 0x7feaa80b0d50 con 0x7feaa814be90 2024-03-25T18:11:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.193+0000 7feaa67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.193+0000 7feaa67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.193+0000 7feaa67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.193+0000 7feaa67fc640 10 monclient(hunting): my global_id is 4301 2024-03-25T18:11:21.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.193+0000 7feaa67fc640 1 -- 172.21.15.64:0/2988354742 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea90003520 con 0x7feaa814be90 2024-03-25T18:11:21.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.203+0000 7feaa67fc640 1 -- 172.21.15.64:0/2988354742 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1795866828 0 0) 0x7fea90003520 con 0x7feaa814be90 2024-03-25T18:11:21.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.203+0000 7feaa67fc640 1 -- 172.21.15.64:0/2988354742 >> v1:172.21.15.86:6789/0 conn(0x7feaa80b1730 legacy=0x7feaa80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:21.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.203+0000 7feaa67fc640 1 -- 172.21.15.64:0/2988354742 >> v1:172.21.15.64:6789/0 conn(0x7feaa80b4d60 legacy=0x7feaa813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:21.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.203+0000 7feaa67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:11:21.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.203+0000 7feaa67fc640 1 monclient: found mon.noname-b 2024-03-25T18:11:21.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.203+0000 7feaa67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:21.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.203+0000 7feaa67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:11:21.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.203+0000 7feaa67fc640 1 -- 172.21.15.64:0/2988354742 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa80b1030 con 0x7feaa814be90 2024-03-25T18:11:21.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.203+0000 7feaa67fc640 10 monclient: _finish_auth 0 2024-03-25T18:11:21.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.203+0000 7feaa67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:11:21.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.203+0000 7feaa67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:21.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.203+0000 7feaa67fc640 1 -- 172.21.15.64:0/2988354742 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7feaa0001a10 con 0x7feaa814be90 2024-03-25T18:11:21.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.203+0000 7feaadb47640 5 monclient: authenticate success, global_id 4301 2024-03-25T18:11:21.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.203+0000 7feaadb47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:21.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.203+0000 7feaa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:21.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.203+0000 7feaa67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:21.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.203+0000 7feaa67fc640 10 monclient: dump: 2024-03-25T18:11:21.205 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:21.205 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:21.205 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:21.205 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:21.205 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:21.205 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:21.205 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:21.205 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:21.205 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:21.205 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:21.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.203+0000 7feaadb47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:21.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.206+0000 7feaa67fc640 1 -- 172.21.15.64:0/2988354742 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feaa0001b70 con 0x7feaa814be90 2024-03-25T18:11:21.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.206+0000 7feaa67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:21.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.206+0000 7feaa67fc640 1 -- 172.21.15.64:0/2988354742 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7feaa0001f80 con 0x7feaa814be90 2024-03-25T18:11:21.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.206+0000 7feaadb47640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:21.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.206+0000 7feaadb47640 4 set_mon_vals no callback set 2024-03-25T18:11:21.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.206+0000 7feaa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:21.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.206+0000 7feaa67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:21.207 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.206+0000 7feaa67fc640 10 monclient: dump: 2024-03-25T18:11:21.207 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:21.208 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:21.208 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:21.208 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:21.208 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:21.208 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:21.208 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:21.208 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:21.208 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:21.208 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:21.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.207+0000 7feaadb47640 10 monclient: shutdown 2024-03-25T18:11:21.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.207+0000 7feaadb47640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:21.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.207+0000 7feaadb47640 1 -- 172.21.15.64:0/2988354742 >> v1:172.21.15.64:6790/0 conn(0x7feaa814be90 legacy=0x7feaa814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:21.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.207+0000 7feaadb47640 1 -- 172.21.15.64:0/2988354742 shutdown_connections 2024-03-25T18:11:21.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.207+0000 7feaadb47640 1 -- 172.21.15.64:0/2988354742 >> 172.21.15.64:0/2988354742 conn(0x7feaa80617a0 msgr2=0x7feaa8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:21.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.208+0000 7feaadb47640 1 -- 172.21.15.64:0/2988354742 shutdown_connections 2024-03-25T18:11:21.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.208+0000 7feaadb47640 1 -- 172.21.15.64:0/2988354742 wait complete. 2024-03-25T18:11:21.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.208+0000 7feaadb47640 10 monclient: build_initial_monmap 2024-03-25T18:11:21.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.208+0000 7feaadb47640 10 monclient: monmap: 2024-03-25T18:11:21.210 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:11:21.210 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:21.210 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:11:21.210 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:11:21.210 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:21.210 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:11:21.210 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:11:21.210 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:11:21.210 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:11:21.210 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:21.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.209+0000 7feaadb47640 1 librados: starting msgr at 2024-03-25T18:11:21.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.209+0000 7feaadb47640 1 librados: starting objecter 2024-03-25T18:11:21.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.209+0000 7feaadb47640 1 Processor -- start 2024-03-25T18:11:21.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.209+0000 7feaadb47640 1 -- start start 2024-03-25T18:11:21.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.209+0000 7feaadb47640 1 librados: setting wanted keys 2024-03-25T18:11:21.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.209+0000 7feaadb47640 1 librados: calling monclient init 2024-03-25T18:11:21.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.209+0000 7feaadb47640 10 monclient: init 2024-03-25T18:11:21.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.210+0000 7feaadb47640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:21.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.210+0000 7feaadb47640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:11:21.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.210+0000 7feaadb47640 10 monclient(hunting): picked mon.noname-b con 0x7feaa80b1730 addr v1:172.21.15.86:6789/0 2024-03-25T18:11:21.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.210+0000 7feaadb47640 10 monclient(hunting): picked mon.noname-c con 0x7feaa80b4d60 addr v1:172.21.15.64:6790/0 2024-03-25T18:11:21.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.210+0000 7feaadb47640 10 monclient(hunting): picked mon.noname-a con 0x7feaa814be90 addr v1:172.21.15.64:6789/0 2024-03-25T18:11:21.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.210+0000 7feaadb47640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa8137510 con 0x7feaa814be90 2024-03-25T18:11:21.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.210+0000 7feaadb47640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa81381a0 con 0x7feaa80b1730 2024-03-25T18:11:21.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.210+0000 7feaadb47640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa8140a90 con 0x7feaa80b4d60 2024-03-25T18:11:21.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.210+0000 7feaadb47640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:21.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.210+0000 7feaadb47640 10 monclient(hunting): authenticate will time out at 7067.448700s 2024-03-25T18:11:21.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.225+0000 7feaa77fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7feaa80b1730 0x7feaa8140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:59032/0 (socket says 172.21.15.64:59032) 2024-03-25T18:11:21.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.225+0000 7feaa77fe640 1 -- 172.21.15.64:0/3097650871 learned_addr learned my addr 172.21.15.64:0/3097650871 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:11:21.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.225+0000 7fea8ffff640 10 client.?.objecter ms_handle_connect 0x7feaa80b1730 2024-03-25T18:11:21.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.225+0000 7fea8ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:21.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.235+0000 7fea8ffff640 10 client.?.objecter ms_handle_connect 0x7feaa814be90 2024-03-25T18:11:21.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.235+0000 7fea8ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:21.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.237+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407205575 0 0) 0x7feaa81381a0 con 0x7feaa80b1730 2024-03-25T18:11:21.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.237+0000 7fea8ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:21.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.237+0000 7fea8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:21.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.237+0000 7fea8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:21.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.237+0000 7fea8ffff640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:11:21.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.237+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea94004eb0 con 0x7feaa80b1730 2024-03-25T18:11:21.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.240+0000 7fea8ffff640 10 client.?.objecter ms_handle_connect 0x7feaa80b4d60 2024-03-25T18:11:21.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.240+0000 7fea8ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.256+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3083903793 0 0) 0x7fea94004eb0 con 0x7feaa80b1730 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.256+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feaa81381a0 con 0x7feaa80b1730 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.256+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fea98001a30 con 0x7feaa80b1730 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.256+0000 7fea8ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.256+0000 7fea8ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.256+0000 7fea8ffff640 10 monclient(hunting): dump: 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:21.257 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:21.258 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:21.258 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:21.258 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:21.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.256+0000 7fea8ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:21.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.258+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436514183 0 0) 0x7feaa8137510 con 0x7feaa814be90 2024-03-25T18:11:21.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.258+0000 7fea8ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:21.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.258+0000 7fea8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:21.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.258+0000 7fea8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:21.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.258+0000 7fea8ffff640 10 monclient(hunting): my global_id is 4302 2024-03-25T18:11:21.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.258+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea94004eb0 con 0x7feaa814be90 2024-03-25T18:11:21.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.271+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044523903 0 0) 0x7feaa8140a90 con 0x7feaa80b4d60 2024-03-25T18:11:21.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.271+0000 7fea8ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:21.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.271+0000 7fea8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:21.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.271+0000 7fea8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:21.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.271+0000 7fea8ffff640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:11:21.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.271+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaa8137510 con 0x7feaa80b4d60 2024-03-25T18:11:21.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3964776027 0 0) 0x7feaa81381a0 con 0x7feaa80b1730 2024-03-25T18:11:21.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 >> v1:172.21.15.64:6790/0 conn(0x7feaa80b4d60 legacy=0x7feaa8140fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:21.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 >> v1:172.21.15.64:6789/0 conn(0x7feaa814be90 legacy=0x7feaa81e7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:21.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7fea8ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:11:21.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7fea8ffff640 1 monclient: found mon.b 2024-03-25T18:11:21.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7fea8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:21.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7fea8ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:11:21.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa81386c0 con 0x7feaa80b1730 2024-03-25T18:11:21.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7fea8ffff640 10 monclient: _finish_auth 0 2024-03-25T18:11:21.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7fea8ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7fea8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7feaadb47640 5 monclient: authenticate success, global_id 4294 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7feaadb47640 10 monclient: _renew_subs 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7feaadb47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7feaadb47640 1 -- 172.21.15.64:0/3097650871 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa8138970 con 0x7feaa80b1730 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7feaadb47640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7feaadb47640 10 monclient: _renew_subs 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7feaadb47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7feaadb47640 1 -- 172.21.15.64:0/3097650871 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa8137df0 con 0x7feaa80b1730 2024-03-25T18:11:21.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.285+0000 7feaadb47640 1 librados: init done 2024-03-25T18:11:21.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.286+0000 7feaadb47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:21.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.286+0000 7feaadb47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:21.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.286+0000 7feaadb47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:11:21.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.286+0000 7feaadb47640 1 -- 172.21.15.64:0/3097650871 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa80627b0 con 0x7feaa80b1730 2024-03-25T18:11:21.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.562+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea98001cb0 con 0x7feaa80b1730 2024-03-25T18:11:21.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.562+0000 7fea8ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:21.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.562+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fea98003a60 con 0x7feaa80b1730 2024-03-25T18:11:21.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.562+0000 7fea8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:21.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.562+0000 7fea8ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:21.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.562+0000 7fea8ffff640 10 monclient: dump: 2024-03-25T18:11:21.564 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:21.564 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:21.564 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:21.564 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:21.564 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:21.564 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:21.564 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:21.564 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:21.565 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:21.565 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:21.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.562+0000 7feaa5ffb640 4 set_mon_vals no callback set 2024-03-25T18:11:21.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.563+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fea98003c80 con 0x7feaa80b1730 2024-03-25T18:11:21.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.563+0000 7feaa6ffd640 10 monclient: get_auth_request con 0x7fea9406de10 auth_method 0 2024-03-25T18:11:21.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.564+0000 7fea8ffff640 10 client.4294.objecter ms_handle_connect 0x7fea9406de10 2024-03-25T18:11:21.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.568+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fea98054910 con 0x7feaa80b1730 2024-03-25T18:11:21.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.568+0000 7fea8ffff640 10 client.4294.objecter ms_dispatch 0x7feaa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:21.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.568+0000 7fea8ffff640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:21.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.568+0000 7fea8ffff640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:21.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.568+0000 7fea8ffff640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-25T18:11:21.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.568+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea98050250 con 0x7feaa80b1730 2024-03-25T18:11:21.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.568+0000 7fea8ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:21.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.568+0000 7fea8ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:21.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.829+0000 7feaadb47640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:21.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.829+0000 7feaadb47640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:21.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.829+0000 7feaadb47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:11:21.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.829+0000 7feaadb47640 1 -- 172.21.15.64:0/3097650871 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feaa8138330 con 0x7feaa80b1730 2024-03-25T18:11:21.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.849+0000 7fea8ffff640 1 -- 172.21.15.64:0/3097650871 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fea98052350 con 0x7feaa80b1730 2024-03-25T18:11:21.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.849+0000 7fea8ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:21.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.849+0000 7fea8ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:21.850 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:11:21.850 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:21.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.851+0000 7feaadb47640 10 librados: watch_flush enter 2024-03-25T18:11:21.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.851+0000 7feaadb47640 10 librados: watch_flush exit 2024-03-25T18:11:21.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.851+0000 7feaadb47640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-25T18:11:21.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.851+0000 7feaadb47640 10 client.4294.objecter successfully canceled tick 2024-03-25T18:11:21.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.851+0000 7feaadb47640 1 -- 172.21.15.64:0/3097650871 >> v1:172.21.15.64:6800/2915543638 conn(0x7fea9406de10 legacy=0x7fea940702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:21.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.851+0000 7feaadb47640 10 monclient: shutdown 2024-03-25T18:11:21.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.851+0000 7feaadb47640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:21.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.851+0000 7feaadb47640 1 -- 172.21.15.64:0/3097650871 >> v1:172.21.15.86:6789/0 conn(0x7feaa80b1730 legacy=0x7feaa8140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:21.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.851+0000 7feaadb47640 1 -- 172.21.15.64:0/3097650871 shutdown_connections 2024-03-25T18:11:21.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.851+0000 7feaadb47640 1 -- 172.21.15.64:0/3097650871 >> 172.21.15.64:0/3097650871 conn(0x7feaa80617a0 msgr2=0x7feaa813c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:21.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.851+0000 7feaadb47640 1 -- 172.21.15.64:0/3097650871 shutdown_connections 2024-03-25T18:11:21.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.851+0000 7feaadb47640 1 -- 172.21.15.64:0/3097650871 wait complete. 2024-03-25T18:11:21.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:21.851+0000 7feaadb47640 1 librados: shutdown 2024-03-25T18:11:21.862 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:28.671 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:28.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.785+0000 7f9b80016640 10 monclient: get_monmap_and_config 2024-03-25T18:11:28.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.785+0000 7f9b80016640 10 monclient: build_initial_monmap 2024-03-25T18:11:28.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.785+0000 7f9b80016640 10 monclient: monmap: 2024-03-25T18:11:28.786 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:11:28.786 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:28.786 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:11:28.786133+0000 2024-03-25T18:11:28.786 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:11:28.786133+0000 2024-03-25T18:11:28.786 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:28.786 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:28.787 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:11:28.787 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:11:28.787 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:11:28.787 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:28.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.785+0000 7f9b80016640 1 Processor -- start 2024-03-25T18:11:28.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.786+0000 7f9b80016640 1 -- start start 2024-03-25T18:11:28.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.786+0000 7f9b80016640 10 monclient: init 2024-03-25T18:11:28.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.786+0000 7f9b80016640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:28.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.786+0000 7f9b80016640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:28.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.786+0000 7f9b80016640 10 monclient(hunting): picked mon.noname-a con 0x7f9b780a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:11:28.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.786+0000 7f9b80016640 10 monclient(hunting): picked mon.noname-b con 0x7f9b7813f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:11:28.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.786+0000 7f9b80016640 10 monclient(hunting): picked mon.noname-c con 0x7f9b781430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:11:28.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.786+0000 7f9b80016640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b78142520 con 0x7f9b780a98f0 2024-03-25T18:11:28.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.786+0000 7f9b80016640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b780b28a0 con 0x7f9b781430c0 2024-03-25T18:11:28.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.786+0000 7f9b80016640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b780b2a30 con 0x7f9b7813f710 2024-03-25T18:11:28.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.786+0000 7f9b80016640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:28.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.786+0000 7f9b80016640 10 monclient(hunting): authenticate will time out at 7075.025284s 2024-03-25T18:11:28.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.801+0000 7f9b7e5b3640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f9b781430c0 0x7f9b7814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:58520/0 (socket says 172.21.15.64:58520) 2024-03-25T18:11:28.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.801+0000 7f9b7e5b3640 1 -- 172.21.15.64:0/2321956996 learned_addr learned my addr 172.21.15.64:0/2321956996 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:11:28.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.816+0000 7f9b7cdb0640 1 -- 172.21.15.64:0/2321956996 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632402641 0 0) 0x7f9b780b28a0 con 0x7f9b781430c0 2024-03-25T18:11:28.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.816+0000 7f9b7cdb0640 10 monclient(hunting): _negotiate 2024-03-25T18:11:28.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.816+0000 7f9b7cdb0640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:28.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.816+0000 7f9b7cdb0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:28.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.816+0000 7f9b7cdb0640 10 monclient(hunting): my global_id is 4297 2024-03-25T18:11:28.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.816+0000 7f9b7cdb0640 1 -- 172.21.15.64:0/2321956996 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b64003520 con 0x7f9b781430c0 2024-03-25T18:11:28.824 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.824+0000 7f9b7cdb0640 1 -- 172.21.15.64:0/2321956996 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717641121 0 0) 0x7f9b780b2a30 con 0x7f9b7813f710 2024-03-25T18:11:28.824 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.824+0000 7f9b7cdb0640 10 monclient(hunting): _negotiate 2024-03-25T18:11:28.824 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.824+0000 7f9b7cdb0640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:28.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.824+0000 7f9b7cdb0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:28.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.824+0000 7f9b7cdb0640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:11:28.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.824+0000 7f9b7cdb0640 1 -- 172.21.15.64:0/2321956996 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b780b28a0 con 0x7f9b7813f710 2024-03-25T18:11:28.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.831+0000 7f9b7cdb0640 1 -- 172.21.15.64:0/2321956996 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444364388 0 0) 0x7f9b78142520 con 0x7f9b780a98f0 2024-03-25T18:11:28.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.831+0000 7f9b7cdb0640 10 monclient(hunting): _negotiate 2024-03-25T18:11:28.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.831+0000 7f9b7cdb0640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:28.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.831+0000 7f9b7cdb0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:28.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.831+0000 7f9b7cdb0640 10 monclient(hunting): my global_id is 4305 2024-03-25T18:11:28.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.831+0000 7f9b7cdb0640 1 -- 172.21.15.64:0/2321956996 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b780b2a30 con 0x7f9b780a98f0 2024-03-25T18:11:28.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.842+0000 7f9b7cdb0640 1 -- 172.21.15.64:0/2321956996 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3967624116 0 0) 0x7f9b64003520 con 0x7f9b781430c0 2024-03-25T18:11:28.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.842+0000 7f9b7cdb0640 1 -- 172.21.15.64:0/2321956996 >> v1:172.21.15.64:6790/0 conn(0x7f9b7813f710 legacy=0x7f9b781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:28.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.842+0000 7f9b7cdb0640 1 -- 172.21.15.64:0/2321956996 >> v1:172.21.15.64:6789/0 conn(0x7f9b780a98f0 legacy=0x7f9b780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:28.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.842+0000 7f9b7cdb0640 10 monclient: _finish_hunting 0 2024-03-25T18:11:28.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.842+0000 7f9b7cdb0640 1 monclient: found mon.noname-c 2024-03-25T18:11:28.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.842+0000 7f9b7cdb0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:28.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.842+0000 7f9b7cdb0640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:11:28.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.842+0000 7f9b7cdb0640 1 -- 172.21.15.64:0/2321956996 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b780b44c0 con 0x7f9b781430c0 2024-03-25T18:11:28.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.842+0000 7f9b7cdb0640 10 monclient: _finish_auth 0 2024-03-25T18:11:28.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.842+0000 7f9b7cdb0640 10 monclient: _check_auth_tickets 2024-03-25T18:11:28.843 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.842+0000 7f9b7cdb0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:28.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.842+0000 7f9b7cdb0640 1 -- 172.21.15.64:0/2321956996 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9b70001a10 con 0x7f9b781430c0 2024-03-25T18:11:28.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.842+0000 7f9b80016640 5 monclient: authenticate success, global_id 4297 2024-03-25T18:11:28.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.842+0000 7f9b80016640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:28.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.842+0000 7f9b7cdb0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:28.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.842+0000 7f9b7cdb0640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:28.844 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.842+0000 7f9b7cdb0640 10 monclient: dump: 2024-03-25T18:11:28.844 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:28.844 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:28.844 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:28.844 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:28.844 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:28.844 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:28.844 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:28.844 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:28.844 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:28.845 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:28.845 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.842+0000 7f9b80016640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:28.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.852+0000 7f9b7cdb0640 1 -- 172.21.15.64:0/2321956996 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b70001b70 con 0x7f9b781430c0 2024-03-25T18:11:28.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.852+0000 7f9b7cdb0640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:28.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.852+0000 7f9b7cdb0640 1 -- 172.21.15.64:0/2321956996 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9b70001f80 con 0x7f9b781430c0 2024-03-25T18:11:28.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.852+0000 7f9b80016640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:28.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.852+0000 7f9b80016640 4 set_mon_vals no callback set 2024-03-25T18:11:28.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.852+0000 7f9b7cdb0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:28.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.852+0000 7f9b7cdb0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:28.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.852+0000 7f9b7cdb0640 10 monclient: dump: 2024-03-25T18:11:28.853 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:28.853 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:28.853 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:28.854 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:28.854 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:28.854 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:28.854 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:28.854 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:28.854 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:28.854 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:28.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.853+0000 7f9b80016640 10 monclient: shutdown 2024-03-25T18:11:28.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.853+0000 7f9b80016640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:28.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.853+0000 7f9b80016640 1 -- 172.21.15.64:0/2321956996 >> v1:172.21.15.86:6789/0 conn(0x7f9b781430c0 legacy=0x7f9b7814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:28.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.853+0000 7f9b80016640 1 -- 172.21.15.64:0/2321956996 shutdown_connections 2024-03-25T18:11:28.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.853+0000 7f9b80016640 1 -- 172.21.15.64:0/2321956996 >> 172.21.15.64:0/2321956996 conn(0x7f9b780b7ea0 msgr2=0x7f9b780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:28.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.853+0000 7f9b80016640 1 -- 172.21.15.64:0/2321956996 shutdown_connections 2024-03-25T18:11:28.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.853+0000 7f9b80016640 1 -- 172.21.15.64:0/2321956996 wait complete. 2024-03-25T18:11:28.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.854+0000 7f9b80016640 10 monclient: build_initial_monmap 2024-03-25T18:11:28.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.854+0000 7f9b80016640 10 monclient: monmap: 2024-03-25T18:11:28.857 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:11:28.857 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:28.857 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:11:28.857 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:11:28.857 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:28.857 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:11:28.857 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:11:28.857 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:11:28.857 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:11:28.857 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:28.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.854+0000 7f9b80016640 1 librados: starting msgr at 2024-03-25T18:11:28.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.854+0000 7f9b80016640 1 librados: starting objecter 2024-03-25T18:11:28.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.854+0000 7f9b80016640 1 Processor -- start 2024-03-25T18:11:28.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.854+0000 7f9b80016640 1 -- start start 2024-03-25T18:11:28.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.854+0000 7f9b80016640 1 librados: setting wanted keys 2024-03-25T18:11:28.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.854+0000 7f9b80016640 1 librados: calling monclient init 2024-03-25T18:11:28.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.854+0000 7f9b80016640 10 monclient: init 2024-03-25T18:11:28.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.855+0000 7f9b80016640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:28.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.855+0000 7f9b80016640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:28.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.855+0000 7f9b80016640 10 monclient(hunting): picked mon.noname-a con 0x7f9b780a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:11:28.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.855+0000 7f9b80016640 10 monclient(hunting): picked mon.noname-b con 0x7f9b7813f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:11:28.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.855+0000 7f9b80016640 10 monclient(hunting): picked mon.noname-c con 0x7f9b781430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:11:28.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.855+0000 7f9b80016640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7813c2b0 con 0x7f9b780a98f0 2024-03-25T18:11:28.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.855+0000 7f9b80016640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b78136f10 con 0x7f9b7813f710 2024-03-25T18:11:28.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.855+0000 7f9b80016640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b781370a0 con 0x7f9b781430c0 2024-03-25T18:11:28.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.855+0000 7f9b80016640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:28.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.855+0000 7f9b80016640 10 monclient(hunting): authenticate will time out at 7075.093765s 2024-03-25T18:11:28.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.857+0000 7f9b7d5b1640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f9b7813f710 0x7f9b781377d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:58574/0 (socket says 172.21.15.64:58574) 2024-03-25T18:11:28.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.857+0000 7f9b7d5b1640 1 -- 172.21.15.64:0/1710157674 learned_addr learned my addr 172.21.15.64:0/1710157674 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:11:28.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.859+0000 7f9b6e7fc640 10 client.?.objecter ms_handle_connect 0x7f9b7813f710 2024-03-25T18:11:28.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.859+0000 7f9b6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:28.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.861+0000 7f9b6e7fc640 10 client.?.objecter ms_handle_connect 0x7f9b780a98f0 2024-03-25T18:11:28.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.861+0000 7f9b6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:28.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.932+0000 7f9b6e7fc640 10 client.?.objecter ms_handle_connect 0x7f9b781430c0 2024-03-25T18:11:28.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.932+0000 7f9b6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:28.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.941+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744836060 0 0) 0x7f9b781370a0 con 0x7f9b781430c0 2024-03-25T18:11:28.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.941+0000 7f9b6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:28.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.941+0000 7f9b6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:28.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.941+0000 7f9b6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:28.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.941+0000 7f9b6e7fc640 10 monclient(hunting): my global_id is 4310 2024-03-25T18:11:28.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.941+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b5c004eb0 con 0x7f9b781430c0 2024-03-25T18:11:28.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.946+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3178603217 0 0) 0x7f9b7813c2b0 con 0x7f9b780a98f0 2024-03-25T18:11:28.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.946+0000 7f9b6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:28.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.946+0000 7f9b6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:28.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.946+0000 7f9b6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:28.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.946+0000 7f9b6e7fc640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:11:28.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.946+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b781370a0 con 0x7f9b780a98f0 2024-03-25T18:11:28.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.952+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194626896 0 0) 0x7f9b78136f10 con 0x7f9b7813f710 2024-03-25T18:11:28.952 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.952+0000 7f9b6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:28.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.952+0000 7f9b6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:28.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.952+0000 7f9b6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:28.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.952+0000 7f9b6e7fc640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:11:28.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.952+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b7813c2b0 con 0x7f9b7813f710 2024-03-25T18:11:28.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.961+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 883650925 0 0) 0x7f9b5c004eb0 con 0x7f9b781430c0 2024-03-25T18:11:28.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.961+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b78136f10 con 0x7f9b781430c0 2024-03-25T18:11:28.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.961+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9b70001e80 con 0x7f9b781430c0 2024-03-25T18:11:28.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.961+0000 7f9b6e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:28.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.961+0000 7f9b6e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:28.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.961+0000 7f9b6e7fc640 10 monclient(hunting): dump: 2024-03-25T18:11:28.962 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:28.962 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:28.962 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:28.963 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:28.963 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:28.963 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:28.963 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:28.963 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:28.963 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:28.963 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:28.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.961+0000 7f9b6e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:28.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.978+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2464394475 0 0) 0x7f9b7813c2b0 con 0x7f9b7813f710 2024-03-25T18:11:28.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.978+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b5c004eb0 con 0x7f9b7813f710 2024-03-25T18:11:28.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.978+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9b68001930 con 0x7f9b7813f710 2024-03-25T18:11:28.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.978+0000 7f9b6e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:28.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.978+0000 7f9b6e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:28.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.978+0000 7f9b6e7fc640 10 monclient(hunting): dump: 2024-03-25T18:11:28.979 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:28.979 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:28.979 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:28.979 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:28.979 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:28.979 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:28.979 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:28.980 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:28.980 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:28.980 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:28.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 19534959 0 0) 0x7f9b781370a0 con 0x7f9b780a98f0 2024-03-25T18:11:28.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b7813c2b0 con 0x7f9b780a98f0 2024-03-25T18:11:28.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9b74001870 con 0x7f9b780a98f0 2024-03-25T18:11:28.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b6e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:28.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b6e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:28.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b6e7fc640 10 monclient(hunting): dump: 2024-03-25T18:11:28.982 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:28.982 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:28.982 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:28.982 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:28.982 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:28.982 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:28.982 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:28.982 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:28.982 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:28.982 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:28.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1928642489 0 0) 0x7f9b78136f10 con 0x7f9b781430c0 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 >> v1:172.21.15.86:6789/0 conn(0x7f9b7813f710 legacy=0x7f9b781377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 >> v1:172.21.15.64:6789/0 conn(0x7f9b780a98f0 legacy=0x7f9b781d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b6e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b6e7fc640 1 monclient: found mon.c 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b6e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b78137230 con 0x7f9b781430c0 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b6e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b6e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b80016640 5 monclient: authenticate success, global_id 4310 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b80016640 10 monclient: _renew_subs 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b80016640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b80016640 1 -- 172.21.15.64:0/1710157674 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b7813b620 con 0x7f9b781430c0 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b80016640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b80016640 10 monclient: _renew_subs 2024-03-25T18:11:28.983 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b80016640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:11:28.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b80016640 1 -- 172.21.15.64:0/1710157674 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b781338f0 con 0x7f9b781430c0 2024-03-25T18:11:28.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.979+0000 7f9b80016640 1 librados: init done 2024-03-25T18:11:28.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.980+0000 7f9b80016640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:28.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.980+0000 7f9b80016640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:28.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.980+0000 7f9b80016640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:11:28.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:28.980+0000 7f9b80016640 1 -- 172.21.15.64:0/1710157674 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b7813b9c0 con 0x7f9b781430c0 2024-03-25T18:11:29.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.288+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b70002020 con 0x7f9b781430c0 2024-03-25T18:11:29.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.288+0000 7f9b6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:29.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.288+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9b70004e10 con 0x7f9b781430c0 2024-03-25T18:11:29.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.288+0000 7f9b6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:29.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.288+0000 7f9b6e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:29.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.288+0000 7f9b6e7fc640 10 monclient: dump: 2024-03-25T18:11:29.290 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:29.290 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:29.290 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:29.290 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:29.290 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:29.290 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:29.290 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:29.290 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:29.290 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:29.290 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:29.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.288+0000 7f9b6ffff640 4 set_mon_vals no callback set 2024-03-25T18:11:29.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.289+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f9b7001ade0 con 0x7f9b781430c0 2024-03-25T18:11:29.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.290+0000 7f9b7ddb2640 10 monclient: get_auth_request con 0x7f9b5c06e0b0 auth_method 0 2024-03-25T18:11:29.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.290+0000 7f9b6e7fc640 10 client.4310.objecter ms_handle_connect 0x7f9b5c06e0b0 2024-03-25T18:11:29.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.295+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f9b70054880 con 0x7f9b781430c0 2024-03-25T18:11:29.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.295+0000 7f9b6e7fc640 10 client.4310.objecter ms_dispatch 0x7f9b78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:29.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.295+0000 7f9b6e7fc640 3 client.4310.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:29.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.295+0000 7f9b6e7fc640 3 client.4310.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:29.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.296+0000 7f9b6e7fc640 20 client.4310.objecter dump_active .. 0 homeless 2024-03-25T18:11:29.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.296+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b70050390 con 0x7f9b781430c0 2024-03-25T18:11:29.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.296+0000 7f9b6e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:29.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.296+0000 7f9b6e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:29.551 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.550+0000 7f9b80016640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:29.551 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.550+0000 7f9b80016640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:29.551 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.550+0000 7f9b80016640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:11:29.551 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.550+0000 7f9b80016640 1 -- 172.21.15.64:0/1710157674 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b780627b0 con 0x7f9b781430c0 2024-03-25T18:11:29.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.583+0000 7f9b6e7fc640 1 -- 172.21.15.64:0/1710157674 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f9b700522c0 con 0x7f9b781430c0 2024-03-25T18:11:29.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.583+0000 7f9b6e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:29.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.583+0000 7f9b6e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:29.584 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:11:29.584 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:29.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.585+0000 7f9b80016640 10 librados: watch_flush enter 2024-03-25T18:11:29.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.585+0000 7f9b80016640 10 librados: watch_flush exit 2024-03-25T18:11:29.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.585+0000 7f9b80016640 20 client.4310.objecter shutdown clearing up homeless session... 2024-03-25T18:11:29.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.585+0000 7f9b80016640 10 client.4310.objecter successfully canceled tick 2024-03-25T18:11:29.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.585+0000 7f9b80016640 1 -- 172.21.15.64:0/1710157674 >> v1:172.21.15.64:6800/2915543638 conn(0x7f9b5c06e0b0 legacy=0x7f9b5c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:29.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.586+0000 7f9b80016640 10 monclient: shutdown 2024-03-25T18:11:29.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.586+0000 7f9b80016640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:29.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.586+0000 7f9b80016640 1 -- 172.21.15.64:0/1710157674 >> v1:172.21.15.64:6790/0 conn(0x7f9b781430c0 legacy=0x7f9b78133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:29.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.586+0000 7f9b80016640 1 -- 172.21.15.64:0/1710157674 shutdown_connections 2024-03-25T18:11:29.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.586+0000 7f9b80016640 1 -- 172.21.15.64:0/1710157674 >> 172.21.15.64:0/1710157674 conn(0x7f9b780b7ea0 msgr2=0x7f9b78074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:29.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.586+0000 7f9b80016640 1 -- 172.21.15.64:0/1710157674 shutdown_connections 2024-03-25T18:11:29.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.586+0000 7f9b80016640 1 -- 172.21.15.64:0/1710157674 wait complete. 2024-03-25T18:11:29.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:29.586+0000 7f9b80016640 1 librados: shutdown 2024-03-25T18:11:29.597 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:31.529 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:02:14.642 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:31.588 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:02:14.701 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:31.627 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:02:14.740 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:31.630 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:02:14.743 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:31.642 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:02:14.755 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:31.651 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:02:14.730 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:31.658 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:02:14.737 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:31.685 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:02:14.798 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:31.688 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:02:14.767 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:36.404 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:36.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.521+0000 7f148c615640 10 monclient: get_monmap_and_config 2024-03-25T18:11:36.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.521+0000 7f148c615640 10 monclient: build_initial_monmap 2024-03-25T18:11:36.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.521+0000 7f148c615640 10 monclient: monmap: 2024-03-25T18:11:36.522 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:11:36.522 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:36.522 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:11:36.522082+0000 2024-03-25T18:11:36.522 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:11:36.522082+0000 2024-03-25T18:11:36.522 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:36.522 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:36.522 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:11:36.522 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:11:36.522 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:11:36.522 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:36.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.521+0000 7f148c615640 1 Processor -- start 2024-03-25T18:11:36.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.521+0000 7f148c615640 1 -- start start 2024-03-25T18:11:36.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.521+0000 7f148c615640 10 monclient: init 2024-03-25T18:11:36.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.522+0000 7f148c615640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:36.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.522+0000 7f148c615640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:36.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.522+0000 7f148c615640 10 monclient(hunting): picked mon.noname-a con 0x7f14840a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:11:36.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.522+0000 7f148c615640 10 monclient(hunting): picked mon.noname-c con 0x7f148413f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:11:36.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.522+0000 7f148c615640 10 monclient(hunting): picked mon.noname-b con 0x7f14841430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:11:36.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.522+0000 7f148c615640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1484142520 con 0x7f14840a98f0 2024-03-25T18:11:36.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.522+0000 7f148c615640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14840b28a0 con 0x7f148413f710 2024-03-25T18:11:36.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.522+0000 7f148c615640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14840b2a30 con 0x7f14841430c0 2024-03-25T18:11:36.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.522+0000 7f148c615640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:36.525 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.522+0000 7f148c615640 10 monclient(hunting): authenticate will time out at 7082.761350s 2024-03-25T18:11:36.530 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.530+0000 7f1489bb0640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f148413f710 0x7f14841429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:34926/0 (socket says 172.21.15.64:34926) 2024-03-25T18:11:36.530 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.530+0000 7f1489bb0640 1 -- 172.21.15.64:0/3993129869 learned_addr learned my addr 172.21.15.64:0/3993129869 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:11:36.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.544+0000 7f14893af640 1 -- 172.21.15.64:0/3993129869 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8257122 0 0) 0x7f14840b28a0 con 0x7f148413f710 2024-03-25T18:11:36.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.544+0000 7f14893af640 10 monclient(hunting): _negotiate 2024-03-25T18:11:36.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.544+0000 7f14893af640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:36.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.544+0000 7f14893af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:36.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.544+0000 7f14893af640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:11:36.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.544+0000 7f14893af640 1 -- 172.21.15.64:0/3993129869 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1470003520 con 0x7f148413f710 2024-03-25T18:11:36.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.550+0000 7f14893af640 1 -- 172.21.15.64:0/3993129869 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659106674 0 0) 0x7f14840b2a30 con 0x7f14841430c0 2024-03-25T18:11:36.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.550+0000 7f14893af640 10 monclient(hunting): _negotiate 2024-03-25T18:11:36.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.550+0000 7f14893af640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:36.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.550+0000 7f14893af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:36.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.550+0000 7f14893af640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:11:36.551 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.550+0000 7f14893af640 1 -- 172.21.15.64:0/3993129869 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14840b28a0 con 0x7f14841430c0 2024-03-25T18:11:36.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.554+0000 7f14893af640 1 -- 172.21.15.64:0/3993129869 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035365060 0 0) 0x7f1484142520 con 0x7f14840a98f0 2024-03-25T18:11:36.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.554+0000 7f14893af640 10 monclient(hunting): _negotiate 2024-03-25T18:11:36.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.554+0000 7f14893af640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:36.555 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.554+0000 7f14893af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:36.555 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.554+0000 7f14893af640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:11:36.555 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.554+0000 7f14893af640 1 -- 172.21.15.64:0/3993129869 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14840b2a30 con 0x7f14840a98f0 2024-03-25T18:11:36.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.557+0000 7f14893af640 1 -- 172.21.15.64:0/3993129869 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3260271651 0 0) 0x7f14840b28a0 con 0x7f14841430c0 2024-03-25T18:11:36.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.557+0000 7f14893af640 1 -- 172.21.15.64:0/3993129869 >> v1:172.21.15.86:6789/0 conn(0x7f148413f710 legacy=0x7f14841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:36.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.557+0000 7f14893af640 1 -- 172.21.15.64:0/3993129869 >> v1:172.21.15.64:6789/0 conn(0x7f14840a98f0 legacy=0x7f14840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:36.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.557+0000 7f14893af640 10 monclient: _finish_hunting 0 2024-03-25T18:11:36.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.557+0000 7f14893af640 1 monclient: found mon.noname-b 2024-03-25T18:11:36.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.557+0000 7f14893af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:36.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.557+0000 7f14893af640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:11:36.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.557+0000 7f14893af640 1 -- 172.21.15.64:0/3993129869 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14840b44c0 con 0x7f14841430c0 2024-03-25T18:11:36.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.557+0000 7f14893af640 10 monclient: _finish_auth 0 2024-03-25T18:11:36.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.557+0000 7f14893af640 10 monclient: _check_auth_tickets 2024-03-25T18:11:36.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.557+0000 7f14893af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:36.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.557+0000 7f14893af640 1 -- 172.21.15.64:0/3993129869 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f147c001670 con 0x7f14841430c0 2024-03-25T18:11:36.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.557+0000 7f14893af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:36.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.557+0000 7f14893af640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:36.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.557+0000 7f14893af640 10 monclient: dump: 2024-03-25T18:11:36.559 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:36.559 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:36.559 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:36.559 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:36.559 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:36.559 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:36.559 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:36.559 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:36.560 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:36.560 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:36.560 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.557+0000 7f148c615640 5 monclient: authenticate success, global_id 4313 2024-03-25T18:11:36.560 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.557+0000 7f148c615640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:36.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.559+0000 7f14893af640 1 -- 172.21.15.64:0/3993129869 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f147c0017d0 con 0x7f14841430c0 2024-03-25T18:11:36.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.559+0000 7f14893af640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:36.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.559+0000 7f14893af640 1 -- 172.21.15.64:0/3993129869 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f147c001be0 con 0x7f14841430c0 2024-03-25T18:11:36.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.559+0000 7f148c615640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:36.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.559+0000 7f148c615640 4 set_mon_vals no callback set 2024-03-25T18:11:36.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.559+0000 7f14893af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:36.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.559+0000 7f14893af640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:36.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.559+0000 7f14893af640 10 monclient: dump: 2024-03-25T18:11:36.561 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:36.561 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:36.561 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:36.561 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:36.562 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:36.562 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:36.562 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:36.562 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:36.562 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:36.562 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:36.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.560+0000 7f148c615640 10 monclient: shutdown 2024-03-25T18:11:36.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.560+0000 7f148c615640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:36.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.560+0000 7f148c615640 1 -- 172.21.15.64:0/3993129869 >> v1:172.21.15.64:6790/0 conn(0x7f14841430c0 legacy=0x7f148414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:36.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.560+0000 7f148c615640 1 -- 172.21.15.64:0/3993129869 shutdown_connections 2024-03-25T18:11:36.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.560+0000 7f148c615640 1 -- 172.21.15.64:0/3993129869 >> 172.21.15.64:0/3993129869 conn(0x7f14840b7ea0 msgr2=0x7f14840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:36.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.560+0000 7f148c615640 1 -- 172.21.15.64:0/3993129869 shutdown_connections 2024-03-25T18:11:36.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.560+0000 7f148c615640 1 -- 172.21.15.64:0/3993129869 wait complete. 2024-03-25T18:11:36.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.561+0000 7f148c615640 10 monclient: build_initial_monmap 2024-03-25T18:11:36.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.561+0000 7f148c615640 10 monclient: monmap: 2024-03-25T18:11:36.564 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:11:36.564 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:36.564 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:11:36.564 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:11:36.564 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:36.564 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:11:36.564 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:11:36.565 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:11:36.565 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:11:36.565 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:36.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.561+0000 7f148c615640 1 librados: starting msgr at 2024-03-25T18:11:36.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.561+0000 7f148c615640 1 librados: starting objecter 2024-03-25T18:11:36.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.562+0000 7f148c615640 1 Processor -- start 2024-03-25T18:11:36.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.562+0000 7f148c615640 1 -- start start 2024-03-25T18:11:36.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.562+0000 7f148c615640 1 librados: setting wanted keys 2024-03-25T18:11:36.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.562+0000 7f148c615640 1 librados: calling monclient init 2024-03-25T18:11:36.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.562+0000 7f148c615640 10 monclient: init 2024-03-25T18:11:36.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.562+0000 7f148c615640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:36.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.562+0000 7f148c615640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:36.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.562+0000 7f148c615640 10 monclient(hunting): picked mon.noname-a con 0x7f14840a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:11:36.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.562+0000 7f148c615640 10 monclient(hunting): picked mon.noname-c con 0x7f148413f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:11:36.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.562+0000 7f148c615640 10 monclient(hunting): picked mon.noname-b con 0x7f14841430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:11:36.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.562+0000 7f148c615640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1484139330 con 0x7f14840a98f0 2024-03-25T18:11:36.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.563+0000 7f148c615640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14840abe70 con 0x7f14841430c0 2024-03-25T18:11:36.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.563+0000 7f148c615640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14840ac150 con 0x7f148413f710 2024-03-25T18:11:36.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.563+0000 7f148c615640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:36.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.563+0000 7f148c615640 10 monclient(hunting): authenticate will time out at 7082.801492s 2024-03-25T18:11:36.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.576+0000 7f1489bb0640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f148413f710 0x7f14840ac570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:36070/0 (socket says 172.21.15.64:36070) 2024-03-25T18:11:36.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.576+0000 7f1489bb0640 1 -- 172.21.15.64:0/450830993 learned_addr learned my addr 172.21.15.64:0/450830993 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:11:36.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.578+0000 7f147affd640 10 client.?.objecter ms_handle_connect 0x7f148413f710 2024-03-25T18:11:36.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.578+0000 7f147affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:36.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.581+0000 7f147affd640 10 client.?.objecter ms_handle_connect 0x7f14840a98f0 2024-03-25T18:11:36.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.581+0000 7f147affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:36.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.584+0000 7f147affd640 10 client.?.objecter ms_handle_connect 0x7f14841430c0 2024-03-25T18:11:36.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.584+0000 7f147affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:36.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.593+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357575994 0 0) 0x7f14840ac150 con 0x7f148413f710 2024-03-25T18:11:36.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.593+0000 7f147affd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:36.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.593+0000 7f147affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:36.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.593+0000 7f147affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:36.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.593+0000 7f147affd640 10 monclient(hunting): my global_id is 4316 2024-03-25T18:11:36.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.593+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1468004eb0 con 0x7f148413f710 2024-03-25T18:11:36.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.595+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083344237 0 0) 0x7f14840abe70 con 0x7f14841430c0 2024-03-25T18:11:36.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.595+0000 7f147affd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:36.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.595+0000 7f147affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:36.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.595+0000 7f147affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:36.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.595+0000 7f147affd640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:11:36.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.596+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14840ac150 con 0x7f14841430c0 2024-03-25T18:11:36.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.599+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989565843 0 0) 0x7f1484139330 con 0x7f14840a98f0 2024-03-25T18:11:36.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.599+0000 7f147affd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:36.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.599+0000 7f147affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:36.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.599+0000 7f147affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:36.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.599+0000 7f147affd640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:11:36.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.599+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14840abe70 con 0x7f14840a98f0 2024-03-25T18:11:36.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.613+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2772767785 0 0) 0x7f14840abe70 con 0x7f14840a98f0 2024-03-25T18:11:36.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.613+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1484139330 con 0x7f14840a98f0 2024-03-25T18:11:36.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.613+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1480001bb0 con 0x7f14840a98f0 2024-03-25T18:11:36.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.613+0000 7f147affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:36.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.613+0000 7f147affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:36.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.613+0000 7f147affd640 10 monclient(hunting): dump: 2024-03-25T18:11:36.615 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:36.615 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:36.615 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:36.615 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:36.615 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:36.615 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:36.615 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:36.615 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:36.615 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:36.615 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:36.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.613+0000 7f147affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:36.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.613+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1691939728 0 0) 0x7f14840ac150 con 0x7f14841430c0 2024-03-25T18:11:36.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.613+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14840abe70 con 0x7f14841430c0 2024-03-25T18:11:36.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.613+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f147c001ae0 con 0x7f14841430c0 2024-03-25T18:11:36.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.613+0000 7f147affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:36.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.613+0000 7f147affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:36.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.613+0000 7f147affd640 10 monclient(hunting): dump: 2024-03-25T18:11:36.616 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:36.616 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:36.616 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:36.616 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:36.616 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:36.616 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:36.616 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:36.616 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:36.616 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:36.616 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:36.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.623+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2048905623 0 0) 0x7f1468004eb0 con 0x7f148413f710 2024-03-25T18:11:36.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.623+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14840ac150 con 0x7f148413f710 2024-03-25T18:11:36.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.623+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1474001950 con 0x7f148413f710 2024-03-25T18:11:36.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.623+0000 7f147affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:36.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.623+0000 7f147affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:36.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.623+0000 7f147affd640 10 monclient(hunting): dump: 2024-03-25T18:11:36.624 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:36.624 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:36.624 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:36.624 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:36.624 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:36.625 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:36.625 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:36.625 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:36.625 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:36.625 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:36.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3384326921 0 0) 0x7f1484139330 con 0x7f14840a98f0 2024-03-25T18:11:36.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 >> v1:172.21.15.64:6790/0 conn(0x7f148413f710 legacy=0x7f14840ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:36.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 >> v1:172.21.15.86:6789/0 conn(0x7f14841430c0 legacy=0x7f14841e2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:36.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f147affd640 10 monclient: _finish_hunting 0 2024-03-25T18:11:36.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f147affd640 1 monclient: found mon.a 2024-03-25T18:11:36.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f147affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:36.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f147affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:11:36.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14840acc80 con 0x7f14840a98f0 2024-03-25T18:11:36.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f147affd640 10 monclient: _finish_auth 0 2024-03-25T18:11:36.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f147affd640 10 monclient: _check_auth_tickets 2024-03-25T18:11:36.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f147affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:36.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f148c615640 5 monclient: authenticate success, global_id 4314 2024-03-25T18:11:36.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f148c615640 10 monclient: _renew_subs 2024-03-25T18:11:36.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f148c615640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:11:36.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f148c615640 1 -- 172.21.15.64:0/450830993 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1484137e30 con 0x7f14840a98f0 2024-03-25T18:11:36.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f148c615640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:36.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f148c615640 10 monclient: _renew_subs 2024-03-25T18:11:36.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f148c615640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:11:36.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f148c615640 1 -- 172.21.15.64:0/450830993 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1484136880 con 0x7f14840a98f0 2024-03-25T18:11:36.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.625+0000 7f148c615640 1 librados: init done 2024-03-25T18:11:36.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.626+0000 7f148c615640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:36.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.626+0000 7f148c615640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:36.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.626+0000 7f148c615640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:11:36.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.626+0000 7f148c615640 1 -- 172.21.15.64:0/450830993 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14840627b0 con 0x7f14840a98f0 2024-03-25T18:11:36.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.634+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14800020c0 con 0x7f14840a98f0 2024-03-25T18:11:36.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.634+0000 7f147affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:36.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.634+0000 7f1488bae640 4 set_mon_vals no callback set 2024-03-25T18:11:36.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.937+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1480003b70 con 0x7f14840a98f0 2024-03-25T18:11:36.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.937+0000 7f147affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:36.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.937+0000 7f147affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:36.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.937+0000 7f147affd640 10 monclient: dump: 2024-03-25T18:11:36.938 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:36.938 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:36.938 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:36.939 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:36.939 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:36.939 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:36.939 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:36.939 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:36.939 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:36.939 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:36.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.938+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f1480019bc0 con 0x7f14840a98f0 2024-03-25T18:11:36.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.938+0000 7f1489bb0640 10 monclient: get_auth_request con 0x7f146806e0f0 auth_method 0 2024-03-25T18:11:36.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.938+0000 7f147affd640 10 client.4314.objecter ms_handle_connect 0x7f146806e0f0 2024-03-25T18:11:36.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.945+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f1480053970 con 0x7f14840a98f0 2024-03-25T18:11:36.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.945+0000 7f147affd640 10 client.4314.objecter ms_dispatch 0x7f1484000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:36.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.945+0000 7f147affd640 3 client.4314.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:36.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.945+0000 7f147affd640 3 client.4314.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:36.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.946+0000 7f147affd640 20 client.4314.objecter dump_active .. 0 homeless 2024-03-25T18:11:36.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.946+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f148004f2b0 con 0x7f14840a98f0 2024-03-25T18:11:36.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.946+0000 7f147affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:36.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:36.946+0000 7f147affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:37.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.174+0000 7f148c615640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:37.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.174+0000 7f148c615640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:37.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.174+0000 7f148c615640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:11:37.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.174+0000 7f148c615640 1 -- 172.21.15.64:0/450830993 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14841390e0 con 0x7f14840a98f0 2024-03-25T18:11:37.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.204+0000 7f147affd640 1 -- 172.21.15.64:0/450830993 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f14800513b0 con 0x7f14840a98f0 2024-03-25T18:11:37.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.204+0000 7f147affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:37.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.204+0000 7f147affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:37.206 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:11:37.206 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:37.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.207+0000 7f148c615640 10 librados: watch_flush enter 2024-03-25T18:11:37.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.207+0000 7f148c615640 10 librados: watch_flush exit 2024-03-25T18:11:37.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.207+0000 7f148c615640 20 client.4314.objecter shutdown clearing up homeless session... 2024-03-25T18:11:37.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.207+0000 7f148c615640 10 client.4314.objecter successfully canceled tick 2024-03-25T18:11:37.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.207+0000 7f148c615640 1 -- 172.21.15.64:0/450830993 >> v1:172.21.15.64:6800/2915543638 conn(0x7f146806e0f0 legacy=0x7f14680705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:37.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.207+0000 7f148c615640 10 monclient: shutdown 2024-03-25T18:11:37.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.207+0000 7f148c615640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:37.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.207+0000 7f148c615640 1 -- 172.21.15.64:0/450830993 >> v1:172.21.15.64:6789/0 conn(0x7f14840a98f0 legacy=0x7f14841389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:37.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.207+0000 7f148c615640 1 -- 172.21.15.64:0/450830993 shutdown_connections 2024-03-25T18:11:37.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.207+0000 7f148c615640 1 -- 172.21.15.64:0/450830993 >> 172.21.15.64:0/450830993 conn(0x7f14840b7ea0 msgr2=0x7f1484074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:37.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.207+0000 7f148c615640 1 -- 172.21.15.64:0/450830993 shutdown_connections 2024-03-25T18:11:37.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.207+0000 7f148c615640 1 -- 172.21.15.64:0/450830993 wait complete. 2024-03-25T18:11:37.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:37.207+0000 7f148c615640 1 librados: shutdown 2024-03-25T18:11:37.221 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:44.025 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:44.136 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.136+0000 7f9070025640 10 monclient: get_monmap_and_config 2024-03-25T18:11:44.136 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.136+0000 7f9070025640 10 monclient: build_initial_monmap 2024-03-25T18:11:44.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.136+0000 7f9070025640 10 monclient: monmap: 2024-03-25T18:11:44.137 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:11:44.137 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:44.137 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:11:44.137069+0000 2024-03-25T18:11:44.137 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:11:44.137069+0000 2024-03-25T18:11:44.137 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:44.137 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:44.137 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:11:44.137 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:11:44.137 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:11:44.137 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:44.138 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.136+0000 7f9070025640 1 Processor -- start 2024-03-25T18:11:44.138 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.137+0000 7f9070025640 1 -- start start 2024-03-25T18:11:44.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.137+0000 7f9070025640 10 monclient: init 2024-03-25T18:11:44.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.137+0000 7f9070025640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:44.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.137+0000 7f9070025640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:11:44.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.137+0000 7f9070025640 10 monclient(hunting): picked mon.noname-b con 0x7f90680b4d60 addr v1:172.21.15.64:6790/0 2024-03-25T18:11:44.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.137+0000 7f9070025640 10 monclient(hunting): picked mon.noname-a con 0x7f90680b1730 addr v1:172.21.15.64:6789/0 2024-03-25T18:11:44.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.137+0000 7f9070025640 10 monclient(hunting): picked mon.noname-c con 0x7f906814be90 addr v1:172.21.15.86:6789/0 2024-03-25T18:11:44.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.137+0000 7f9070025640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90680af940 con 0x7f90680b1730 2024-03-25T18:11:44.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.137+0000 7f9070025640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90680b0a70 con 0x7f906814be90 2024-03-25T18:11:44.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.137+0000 7f9070025640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90680b0d50 con 0x7f90680b4d60 2024-03-25T18:11:44.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.137+0000 7f9070025640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:44.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.137+0000 7f9070025640 10 monclient(hunting): authenticate will time out at 7090.376089s 2024-03-25T18:11:44.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.144+0000 7f906ddc1640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f90680b4d60 0x7f906813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:40658/0 (socket says 172.21.15.64:40658) 2024-03-25T18:11:44.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.144+0000 7f906ddc1640 1 -- 172.21.15.64:0/1765333352 learned_addr learned my addr 172.21.15.64:0/1765333352 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:11:44.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.167+0000 7f906cdbf640 1 -- 172.21.15.64:0/1765333352 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938191529 0 0) 0x7f90680af940 con 0x7f90680b1730 2024-03-25T18:11:44.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.168+0000 7f906cdbf640 10 monclient(hunting): _negotiate 2024-03-25T18:11:44.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.168+0000 7f906cdbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:44.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.168+0000 7f906cdbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:44.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.168+0000 7f906cdbf640 10 monclient(hunting): my global_id is 4317 2024-03-25T18:11:44.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.168+0000 7f906cdbf640 1 -- 172.21.15.64:0/1765333352 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9054003520 con 0x7f90680b1730 2024-03-25T18:11:44.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.171+0000 7f906cdbf640 1 -- 172.21.15.64:0/1765333352 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264931312 0 0) 0x7f90680b0a70 con 0x7f906814be90 2024-03-25T18:11:44.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.171+0000 7f906cdbf640 10 monclient(hunting): _negotiate 2024-03-25T18:11:44.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.171+0000 7f906cdbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:44.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.171+0000 7f906cdbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:44.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.171+0000 7f906cdbf640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:11:44.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.171+0000 7f906cdbf640 1 -- 172.21.15.64:0/1765333352 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90680af940 con 0x7f906814be90 2024-03-25T18:11:44.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.184+0000 7f906cdbf640 1 -- 172.21.15.64:0/1765333352 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189346554 0 0) 0x7f90680b0d50 con 0x7f90680b4d60 2024-03-25T18:11:44.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.185+0000 7f906cdbf640 10 monclient(hunting): _negotiate 2024-03-25T18:11:44.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.185+0000 7f906cdbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:44.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.185+0000 7f906cdbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:44.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.185+0000 7f906cdbf640 10 monclient(hunting): my global_id is 4319 2024-03-25T18:11:44.185 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.185+0000 7f906cdbf640 1 -- 172.21.15.64:0/1765333352 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90680b0a70 con 0x7f90680b4d60 2024-03-25T18:11:44.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.191+0000 7f906cdbf640 1 -- 172.21.15.64:0/1765333352 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2617082056 0 0) 0x7f90680af940 con 0x7f906814be90 2024-03-25T18:11:44.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.191+0000 7f906cdbf640 1 -- 172.21.15.64:0/1765333352 >> v1:172.21.15.64:6790/0 conn(0x7f90680b4d60 legacy=0x7f906813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:44.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.191+0000 7f906cdbf640 1 -- 172.21.15.64:0/1765333352 >> v1:172.21.15.64:6789/0 conn(0x7f90680b1730 legacy=0x7f90680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:44.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.191+0000 7f906cdbf640 10 monclient: _finish_hunting 0 2024-03-25T18:11:44.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.191+0000 7f906cdbf640 1 monclient: found mon.noname-c 2024-03-25T18:11:44.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.191+0000 7f906cdbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:44.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.191+0000 7f906cdbf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.191+0000 7f906cdbf640 1 -- 172.21.15.64:0/1765333352 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90680b1030 con 0x7f906814be90 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.191+0000 7f906cdbf640 10 monclient: _finish_auth 0 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.191+0000 7f906cdbf640 10 monclient: _check_auth_tickets 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.191+0000 7f906cdbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.191+0000 7f906cdbf640 1 -- 172.21.15.64:0/1765333352 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f90600019f0 con 0x7f906814be90 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.191+0000 7f9070025640 5 monclient: authenticate success, global_id 4309 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.191+0000 7f9070025640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.191+0000 7f906cdbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.191+0000 7f906cdbf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.191+0000 7f906cdbf640 10 monclient: dump: 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:44.194 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:44.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.191+0000 7f9070025640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:44.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.198+0000 7f906cdbf640 1 -- 172.21.15.64:0/1765333352 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9060001b50 con 0x7f906814be90 2024-03-25T18:11:44.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.198+0000 7f906cdbf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:44.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.198+0000 7f906cdbf640 1 -- 172.21.15.64:0/1765333352 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9060001f60 con 0x7f906814be90 2024-03-25T18:11:44.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.198+0000 7f906cdbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:44.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.198+0000 7f906cdbf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:44.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.198+0000 7f906cdbf640 10 monclient: dump: 2024-03-25T18:11:44.199 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:44.200 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:44.200 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:44.200 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:44.200 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:44.200 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:44.200 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:44.200 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:44.200 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:44.200 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:44.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.198+0000 7f9070025640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:44.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.198+0000 7f9070025640 4 set_mon_vals no callback set 2024-03-25T18:11:44.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.199+0000 7f9070025640 10 monclient: shutdown 2024-03-25T18:11:44.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.199+0000 7f9070025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:44.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.199+0000 7f9070025640 1 -- 172.21.15.64:0/1765333352 >> v1:172.21.15.86:6789/0 conn(0x7f906814be90 legacy=0x7f906814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:44.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.200+0000 7f9070025640 1 -- 172.21.15.64:0/1765333352 shutdown_connections 2024-03-25T18:11:44.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.200+0000 7f9070025640 1 -- 172.21.15.64:0/1765333352 >> 172.21.15.64:0/1765333352 conn(0x7f90680617a0 msgr2=0x7f9068061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:44.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.200+0000 7f9070025640 1 -- 172.21.15.64:0/1765333352 shutdown_connections 2024-03-25T18:11:44.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.200+0000 7f9070025640 1 -- 172.21.15.64:0/1765333352 wait complete. 2024-03-25T18:11:44.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.200+0000 7f9070025640 10 monclient: build_initial_monmap 2024-03-25T18:11:44.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.200+0000 7f9070025640 10 monclient: monmap: 2024-03-25T18:11:44.202 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:11:44.202 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:44.202 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:11:44.202 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:11:44.202 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:44.202 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:11:44.202 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:11:44.203 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:11:44.203 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:11:44.203 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:44.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.201+0000 7f9070025640 1 librados: starting msgr at 2024-03-25T18:11:44.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.201+0000 7f9070025640 1 librados: starting objecter 2024-03-25T18:11:44.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.201+0000 7f9070025640 1 Processor -- start 2024-03-25T18:11:44.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.201+0000 7f9070025640 1 -- start start 2024-03-25T18:11:44.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.201+0000 7f9070025640 1 librados: setting wanted keys 2024-03-25T18:11:44.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.201+0000 7f9070025640 1 librados: calling monclient init 2024-03-25T18:11:44.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.201+0000 7f9070025640 10 monclient: init 2024-03-25T18:11:44.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.201+0000 7f9070025640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:44.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.202+0000 7f9070025640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:11:44.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.202+0000 7f9070025640 10 monclient(hunting): picked mon.noname-b con 0x7f90680b1730 addr v1:172.21.15.86:6789/0 2024-03-25T18:11:44.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.202+0000 7f9070025640 10 monclient(hunting): picked mon.noname-a con 0x7f90680b4d60 addr v1:172.21.15.64:6789/0 2024-03-25T18:11:44.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.202+0000 7f9070025640 10 monclient(hunting): picked mon.noname-c con 0x7f906814be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:11:44.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.202+0000 7f9070025640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90681406f0 con 0x7f90680b4d60 2024-03-25T18:11:44.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.202+0000 7f9070025640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9068140380 con 0x7f90680b1730 2024-03-25T18:11:44.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.202+0000 7f9070025640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9068144ce0 con 0x7f906814be90 2024-03-25T18:11:44.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.202+0000 7f9070025640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:44.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.202+0000 7f9070025640 10 monclient(hunting): authenticate will time out at 7090.440637s 2024-03-25T18:11:44.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.214+0000 7f906d5c0640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f90680b4d60 0x7f90681d7620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:59738/0 (socket says 172.21.15.64:59738) 2024-03-25T18:11:44.215 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.214+0000 7f906d5c0640 1 -- 172.21.15.64:0/851968648 learned_addr learned my addr 172.21.15.64:0/851968648 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:11:44.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.215+0000 7f905e7fc640 10 client.?.objecter ms_handle_connect 0x7f90680b4d60 2024-03-25T18:11:44.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.215+0000 7f905e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:44.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.226+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336282032 0 0) 0x7f90681406f0 con 0x7f90680b4d60 2024-03-25T18:11:44.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.226+0000 7f905e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:44.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.226+0000 7f905e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:44.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.226+0000 7f905e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:44.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.226+0000 7f905e7fc640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:11:44.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.226+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f904c004eb0 con 0x7f90680b4d60 2024-03-25T18:11:44.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.231+0000 7f905e7fc640 10 client.?.objecter ms_handle_connect 0x7f90680b1730 2024-03-25T18:11:44.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.231+0000 7f905e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:44.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.239+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 426520714 0 0) 0x7f9068140380 con 0x7f90680b1730 2024-03-25T18:11:44.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.239+0000 7f905e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:44.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.239+0000 7f905e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:44.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.239+0000 7f905e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:44.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.239+0000 7f905e7fc640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:11:44.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.239+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90681406f0 con 0x7f90680b1730 2024-03-25T18:11:44.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.244+0000 7f905e7fc640 10 client.?.objecter ms_handle_connect 0x7f906814be90 2024-03-25T18:11:44.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.244+0000 7f905e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:44.251 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.250+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 720488433 0 0) 0x7f90681406f0 con 0x7f90680b1730 2024-03-25T18:11:44.251 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.250+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9068140380 con 0x7f90680b1730 2024-03-25T18:11:44.251 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.250+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f90640018f0 con 0x7f90680b1730 2024-03-25T18:11:44.252 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.250+0000 7f905e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:44.252 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.250+0000 7f905e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:44.252 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.250+0000 7f905e7fc640 10 monclient(hunting): dump: 2024-03-25T18:11:44.252 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:44.252 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:44.252 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:44.252 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:44.252 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:44.252 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:44.252 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:44.252 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:44.252 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:44.252 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:44.252 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.251+0000 7f905e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.255+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3200937969 0 0) 0x7f904c004eb0 con 0x7f90680b4d60 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.255+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90681406f0 con 0x7f90680b4d60 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.255+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9058001990 con 0x7f90680b4d60 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.255+0000 7f905e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.255+0000 7f905e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.255+0000 7f905e7fc640 10 monclient(hunting): dump: 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:44.256 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:44.257 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:44.257 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:44.257 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:44.257 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:44.257 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:44.257 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:44.257 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:44.257 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:44.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.258+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317185314 0 0) 0x7f9068144ce0 con 0x7f906814be90 2024-03-25T18:11:44.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.258+0000 7f905e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:44.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.258+0000 7f905e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:44.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.258+0000 7f905e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:44.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.258+0000 7f905e7fc640 10 monclient(hunting): my global_id is 4322 2024-03-25T18:11:44.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.258+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f904c004eb0 con 0x7f906814be90 2024-03-25T18:11:44.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2688749438 0 0) 0x7f90681406f0 con 0x7f90680b4d60 2024-03-25T18:11:44.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 >> v1:172.21.15.64:6790/0 conn(0x7f906814be90 legacy=0x7f90681d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:44.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 >> v1:172.21.15.86:6789/0 conn(0x7f90680b1730 legacy=0x7f9068131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:44.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f905e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:11:44.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f905e7fc640 1 monclient: found mon.a 2024-03-25T18:11:44.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f905e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:44.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f905e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:11:44.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9068132520 con 0x7f90680b4d60 2024-03-25T18:11:44.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f905e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:11:44.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f905e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:11:44.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f905e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:44.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f9070025640 5 monclient: authenticate success, global_id 4320 2024-03-25T18:11:44.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f9070025640 10 monclient: _renew_subs 2024-03-25T18:11:44.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f9070025640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:11:44.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f9070025640 1 -- 172.21.15.64:0/851968648 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9068132770 con 0x7f90680b4d60 2024-03-25T18:11:44.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f9070025640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:44.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f9070025640 10 monclient: _renew_subs 2024-03-25T18:11:44.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f9070025640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:11:44.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f9070025640 1 -- 172.21.15.64:0/851968648 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f906813e130 con 0x7f90680b4d60 2024-03-25T18:11:44.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.275+0000 7f9070025640 1 librados: init done 2024-03-25T18:11:44.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.276+0000 7f9070025640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:44.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.276+0000 7f9070025640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:44.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.276+0000 7f9070025640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:11:44.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.276+0000 7f9070025640 1 -- 172.21.15.64:0/851968648 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9068140510 con 0x7f90680b4d60 2024-03-25T18:11:44.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.590+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9058003610 con 0x7f90680b4d60 2024-03-25T18:11:44.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.590+0000 7f905e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:44.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.590+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f90580039e0 con 0x7f90680b4d60 2024-03-25T18:11:44.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.590+0000 7f905e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:44.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.590+0000 7f905e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:44.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.590+0000 7f905e7fc640 10 monclient: dump: 2024-03-25T18:11:44.591 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:44.591 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:44.591 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:44.592 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:44.592 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:44.592 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:44.592 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:44.592 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:44.592 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:44.592 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:44.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.590+0000 7f905ffff640 4 set_mon_vals no callback set 2024-03-25T18:11:44.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.591+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f9058003c00 con 0x7f90680b4d60 2024-03-25T18:11:44.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.592+0000 7f906ddc1640 10 monclient: get_auth_request con 0x7f904c06df00 auth_method 0 2024-03-25T18:11:44.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.592+0000 7f905e7fc640 10 client.4320.objecter ms_handle_connect 0x7f904c06df00 2024-03-25T18:11:44.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.600+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f9058054800 con 0x7f90680b4d60 2024-03-25T18:11:44.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.600+0000 7f905e7fc640 10 client.4320.objecter ms_dispatch 0x7f9068000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:44.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.600+0000 7f905e7fc640 3 client.4320.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:44.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.600+0000 7f905e7fc640 3 client.4320.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:44.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.600+0000 7f905e7fc640 20 client.4320.objecter dump_active .. 0 homeless 2024-03-25T18:11:44.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.600+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9058050310 con 0x7f90680b4d60 2024-03-25T18:11:44.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.600+0000 7f905e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:44.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.600+0000 7f905e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:44.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.848+0000 7f9070025640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:44.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.848+0000 7f9070025640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:44.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.848+0000 7f9070025640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:11:44.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.848+0000 7f9070025640 1 -- 172.21.15.64:0/851968648 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90680627b0 con 0x7f90680b4d60 2024-03-25T18:11:44.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.896+0000 7f905e7fc640 1 -- 172.21.15.64:0/851968648 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f9058052240 con 0x7f90680b4d60 2024-03-25T18:11:44.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.896+0000 7f905e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:44.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.896+0000 7f905e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:44.897 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:11:44.897 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:44.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.898+0000 7f9070025640 10 librados: watch_flush enter 2024-03-25T18:11:44.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.898+0000 7f9070025640 10 librados: watch_flush exit 2024-03-25T18:11:44.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.898+0000 7f9070025640 20 client.4320.objecter shutdown clearing up homeless session... 2024-03-25T18:11:44.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.898+0000 7f9070025640 10 client.4320.objecter successfully canceled tick 2024-03-25T18:11:44.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.898+0000 7f9070025640 1 -- 172.21.15.64:0/851968648 >> v1:172.21.15.64:6800/2915543638 conn(0x7f904c06df00 legacy=0x7f904c0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:44.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.898+0000 7f9070025640 10 monclient: shutdown 2024-03-25T18:11:44.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.898+0000 7f9070025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:44.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.898+0000 7f9070025640 1 -- 172.21.15.64:0/851968648 >> v1:172.21.15.64:6789/0 conn(0x7f90680b4d60 legacy=0x7f90681d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:44.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.898+0000 7f9070025640 1 -- 172.21.15.64:0/851968648 shutdown_connections 2024-03-25T18:11:44.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.898+0000 7f9070025640 1 -- 172.21.15.64:0/851968648 >> 172.21.15.64:0/851968648 conn(0x7f90680617a0 msgr2=0x7f906813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:44.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.898+0000 7f9070025640 1 -- 172.21.15.64:0/851968648 shutdown_connections 2024-03-25T18:11:44.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.898+0000 7f9070025640 1 -- 172.21.15.64:0/851968648 wait complete. 2024-03-25T18:11:44.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:44.898+0000 7f9070025640 1 librados: shutdown 2024-03-25T18:11:44.910 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:51.572 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:02:34.685 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:51.625 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:02:34.738 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:51.651 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:02:34.764 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:51.679 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:02:34.792 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:51.687 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:02:34.766 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:51.693 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:02:34.806 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:51.694 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:02:34.773 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:51.717 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:51.731 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:02:34.844 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:51.735 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:02:34.813 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:51.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.827+0000 7f2a4700d640 10 monclient: get_monmap_and_config 2024-03-25T18:11:51.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.827+0000 7f2a4700d640 10 monclient: build_initial_monmap 2024-03-25T18:11:51.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.827+0000 7f2a4700d640 10 monclient: monmap: 2024-03-25T18:11:51.829 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:11:51.829 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:51.829 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:11:51.829728+0000 2024-03-25T18:11:51.829 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:11:51.829728+0000 2024-03-25T18:11:51.829 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:51.830 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:51.830 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:11:51.830 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:11:51.830 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:11:51.830 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:51.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.828+0000 7f2a4700d640 1 Processor -- start 2024-03-25T18:11:51.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.828+0000 7f2a4700d640 1 -- start start 2024-03-25T18:11:51.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.828+0000 7f2a4700d640 10 monclient: init 2024-03-25T18:11:51.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.829+0000 7f2a4700d640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:51.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.829+0000 7f2a4700d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:11:51.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.829+0000 7f2a4700d640 10 monclient(hunting): picked mon.noname-b con 0x7f2a40061830 addr v1:172.21.15.64:6790/0 2024-03-25T18:11:51.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.829+0000 7f2a4700d640 10 monclient(hunting): picked mon.noname-c con 0x7f2a4013db20 addr v1:172.21.15.86:6789/0 2024-03-25T18:11:51.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.829+0000 7f2a4700d640 10 monclient(hunting): picked mon.noname-a con 0x7f2a400b0510 addr v1:172.21.15.64:6789/0 2024-03-25T18:11:51.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.830+0000 7f2a4700d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a4006eb80 con 0x7f2a400b0510 2024-03-25T18:11:51.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.830+0000 7f2a4700d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a40072b10 con 0x7f2a4013db20 2024-03-25T18:11:51.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.830+0000 7f2a4700d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a40072df0 con 0x7f2a40061830 2024-03-25T18:11:51.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.830+0000 7f2a4700d640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:51.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.830+0000 7f2a4700d640 10 monclient(hunting): authenticate will time out at 7098.069535s 2024-03-25T18:11:51.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.839+0000 7f2a3ffff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2a4013db20 0x7f2a400afe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:60284/0 (socket says 172.21.15.64:60284) 2024-03-25T18:11:51.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.839+0000 7f2a3ffff640 1 -- 172.21.15.64:0/4070688393 learned_addr learned my addr 172.21.15.64:0/4070688393 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:11:51.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.854+0000 7f2a3f7fe640 1 -- 172.21.15.64:0/4070688393 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43260351 0 0) 0x7f2a40072b10 con 0x7f2a4013db20 2024-03-25T18:11:51.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.854+0000 7f2a3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:51.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.854+0000 7f2a3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:51.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.854+0000 7f2a3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:51.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.854+0000 7f2a3f7fe640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:11:51.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.854+0000 7f2a3f7fe640 1 -- 172.21.15.64:0/4070688393 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a30003520 con 0x7f2a4013db20 2024-03-25T18:11:51.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.876+0000 7f2a3f7fe640 1 -- 172.21.15.64:0/4070688393 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727845423 0 0) 0x7f2a40072df0 con 0x7f2a40061830 2024-03-25T18:11:51.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:51.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:51.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:51.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 10 monclient(hunting): my global_id is 4325 2024-03-25T18:11:51.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 1 -- 172.21.15.64:0/4070688393 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a40072b10 con 0x7f2a40061830 2024-03-25T18:11:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 1 -- 172.21.15.64:0/4070688393 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2936610524 0 0) 0x7f2a30003520 con 0x7f2a4013db20 2024-03-25T18:11:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 1 -- 172.21.15.64:0/4070688393 >> v1:172.21.15.64:6790/0 conn(0x7f2a40061830 legacy=0x7f2a4013d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 1 -- 172.21.15.64:0/4070688393 >> v1:172.21.15.64:6789/0 conn(0x7f2a400b0510 legacy=0x7f2a4014ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:11:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:11:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:11:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 1 -- 172.21.15.64:0/4070688393 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a400b0980 con 0x7f2a4013db20 2024-03-25T18:11:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:11:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:11:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:51.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 1 -- 172.21.15.64:0/4070688393 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2a2c001690 con 0x7f2a4013db20 2024-03-25T18:11:51.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:51.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:51.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a3f7fe640 10 monclient: dump: 2024-03-25T18:11:51.881 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:51.881 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:51.881 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:51.881 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:51.881 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:51.881 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:51.881 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:51.881 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:51.881 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:51.881 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:51.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a4700d640 5 monclient: authenticate success, global_id 4315 2024-03-25T18:11:51.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.877+0000 7f2a4700d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:51.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.891+0000 7f2a3f7fe640 1 -- 172.21.15.64:0/4070688393 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a2c0017f0 con 0x7f2a4013db20 2024-03-25T18:11:51.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.891+0000 7f2a3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:51.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.891+0000 7f2a3f7fe640 1 -- 172.21.15.64:0/4070688393 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2a2c001b80 con 0x7f2a4013db20 2024-03-25T18:11:51.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.891+0000 7f2a4700d640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:51.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.891+0000 7f2a4700d640 4 set_mon_vals no callback set 2024-03-25T18:11:51.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.891+0000 7f2a3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:51.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.891+0000 7f2a3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:51.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.891+0000 7f2a3f7fe640 10 monclient: dump: 2024-03-25T18:11:51.894 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:51.894 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:51.894 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:51.894 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:51.894 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:51.894 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:51.894 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:51.894 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:51.894 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:51.894 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:51.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.892+0000 7f2a4700d640 10 monclient: shutdown 2024-03-25T18:11:51.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.892+0000 7f2a4700d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.892+0000 7f2a4700d640 1 -- 172.21.15.64:0/4070688393 >> v1:172.21.15.86:6789/0 conn(0x7f2a4013db20 legacy=0x7f2a400afe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.892+0000 7f2a4700d640 1 -- 172.21.15.64:0/4070688393 shutdown_connections 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.892+0000 7f2a4700d640 1 -- 172.21.15.64:0/4070688393 >> 172.21.15.64:0/4070688393 conn(0x7f2a400b0ef0 msgr2=0x7f2a400b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.893+0000 7f2a4700d640 1 -- 172.21.15.64:0/4070688393 shutdown_connections 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.893+0000 7f2a4700d640 1 -- 172.21.15.64:0/4070688393 wait complete. 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.893+0000 7f2a4700d640 10 monclient: build_initial_monmap 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.893+0000 7f2a4700d640 10 monclient: monmap: 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:11:51.896 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:11:51.897 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:11:51.897 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:51.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.894+0000 7f2a4700d640 1 librados: starting msgr at 2024-03-25T18:11:51.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.894+0000 7f2a4700d640 1 librados: starting objecter 2024-03-25T18:11:51.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.894+0000 7f2a4700d640 1 Processor -- start 2024-03-25T18:11:51.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.894+0000 7f2a4700d640 1 -- start start 2024-03-25T18:11:51.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.894+0000 7f2a4700d640 1 librados: setting wanted keys 2024-03-25T18:11:51.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.894+0000 7f2a4700d640 1 librados: calling monclient init 2024-03-25T18:11:51.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.894+0000 7f2a4700d640 10 monclient: init 2024-03-25T18:11:51.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.895+0000 7f2a4700d640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:51.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.895+0000 7f2a4700d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:51.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.895+0000 7f2a4700d640 10 monclient(hunting): picked mon.noname-a con 0x7f2a40061830 addr v1:172.21.15.64:6789/0 2024-03-25T18:11:51.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.895+0000 7f2a4700d640 10 monclient(hunting): picked mon.noname-c con 0x7f2a400b0510 addr v1:172.21.15.64:6790/0 2024-03-25T18:11:51.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.895+0000 7f2a4700d640 10 monclient(hunting): picked mon.noname-b con 0x7f2a4013db20 addr v1:172.21.15.86:6789/0 2024-03-25T18:11:51.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.895+0000 7f2a4700d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a401db5d0 con 0x7f2a40061830 2024-03-25T18:11:51.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.895+0000 7f2a4700d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a401db890 con 0x7f2a4013db20 2024-03-25T18:11:51.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.895+0000 7f2a4700d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a40120ed0 con 0x7f2a400b0510 2024-03-25T18:11:51.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.895+0000 7f2a4700d640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:51.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.895+0000 7f2a4700d640 10 monclient(hunting): authenticate will time out at 7098.134890s 2024-03-25T18:11:51.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.910+0000 7f2a3ffff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f2a400b0510 0x7f2a4011eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:54318/0 (socket says 172.21.15.64:54318) 2024-03-25T18:11:51.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.910+0000 7f2a3ffff640 1 -- 172.21.15.64:0/801706442 learned_addr learned my addr 172.21.15.64:0/801706442 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:11:51.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.914+0000 7f2a3d7fa640 10 client.?.objecter ms_handle_connect 0x7f2a400b0510 2024-03-25T18:11:51.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.914+0000 7f2a3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:51.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.915+0000 7f2a3d7fa640 10 client.?.objecter ms_handle_connect 0x7f2a4013db20 2024-03-25T18:11:51.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.915+0000 7f2a3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.923+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2669454664 0 0) 0x7f2a401db890 con 0x7f2a4013db20 2024-03-25T18:11:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.923+0000 7f2a3d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.923+0000 7f2a3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.923+0000 7f2a3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.923+0000 7f2a3d7fa640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:11:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.923+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a24004eb0 con 0x7f2a4013db20 2024-03-25T18:11:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.923+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350334517 0 0) 0x7f2a40120ed0 con 0x7f2a400b0510 2024-03-25T18:11:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.923+0000 7f2a3d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.923+0000 7f2a3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.923+0000 7f2a3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:51.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.923+0000 7f2a3d7fa640 10 monclient(hunting): my global_id is 4328 2024-03-25T18:11:51.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.923+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a401db890 con 0x7f2a400b0510 2024-03-25T18:11:51.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.923+0000 7f2a3d7fa640 10 client.?.objecter ms_handle_connect 0x7f2a40061830 2024-03-25T18:11:51.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.923+0000 7f2a3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:51.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.936+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620759664 0 0) 0x7f2a401db5d0 con 0x7f2a40061830 2024-03-25T18:11:51.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.936+0000 7f2a3d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:51.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.936+0000 7f2a3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:51.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.936+0000 7f2a3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:51.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.936+0000 7f2a3d7fa640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:11:51.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.936+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a40120ed0 con 0x7f2a40061830 2024-03-25T18:11:51.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.939+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2679751029 0 0) 0x7f2a401db890 con 0x7f2a400b0510 2024-03-25T18:11:51.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.939+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a401db5d0 con 0x7f2a400b0510 2024-03-25T18:11:51.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.939+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2a2c001b00 con 0x7f2a400b0510 2024-03-25T18:11:51.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.939+0000 7f2a3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:51.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.939+0000 7f2a3d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:51.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.939+0000 7f2a3d7fa640 10 monclient(hunting): dump: 2024-03-25T18:11:51.942 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:51.942 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:51.942 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:51.942 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:51.942 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:51.942 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:51.942 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:51.942 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:51.942 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:51.942 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:51.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.940+0000 7f2a3d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:51.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.943+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1659930586 0 0) 0x7f2a24004eb0 con 0x7f2a4013db20 2024-03-25T18:11:51.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.943+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a401db890 con 0x7f2a4013db20 2024-03-25T18:11:51.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.943+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2a38001c30 con 0x7f2a4013db20 2024-03-25T18:11:51.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.943+0000 7f2a3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:51.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.943+0000 7f2a3d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:51.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.943+0000 7f2a3d7fa640 10 monclient(hunting): dump: 2024-03-25T18:11:51.945 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:51.945 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:51.945 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:51.945 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:51.945 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:51.945 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:51.946 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:51.946 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:51.946 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:51.946 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:51.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.948+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3505591416 0 0) 0x7f2a40120ed0 con 0x7f2a40061830 2024-03-25T18:11:51.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.949+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a24004eb0 con 0x7f2a40061830 2024-03-25T18:11:51.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.949+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2a34001930 con 0x7f2a40061830 2024-03-25T18:11:51.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.949+0000 7f2a3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:51.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.949+0000 7f2a3d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:51.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.949+0000 7f2a3d7fa640 10 monclient(hunting): dump: 2024-03-25T18:11:51.953 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:51.953 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:51.953 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:51.953 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:51.953 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:51.953 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:51.953 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:51.953 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:51.953 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:51.953 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:51.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1333566394 0 0) 0x7f2a401db5d0 con 0x7f2a400b0510 2024-03-25T18:11:51.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 >> v1:172.21.15.86:6789/0 conn(0x7f2a4013db20 legacy=0x7f2a4011f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:51.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 >> v1:172.21.15.64:6789/0 conn(0x7f2a40061830 legacy=0x7f2a401daec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:51.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a3d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:11:51.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a3d7fa640 1 monclient: found mon.c 2024-03-25T18:11:51.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a3d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:51.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a3d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:11:51.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4013b930 con 0x7f2a400b0510 2024-03-25T18:11:51.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a3d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:11:51.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a3d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:11:51.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:51.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a4700d640 5 monclient: authenticate success, global_id 4328 2024-03-25T18:11:51.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a4700d640 10 monclient: _renew_subs 2024-03-25T18:11:51.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a4700d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:11:51.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a4700d640 1 -- 172.21.15.64:0/801706442 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a4013bbe0 con 0x7f2a400b0510 2024-03-25T18:11:51.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a4700d640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:51.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a4700d640 10 monclient: _renew_subs 2024-03-25T18:11:51.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a4700d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:11:51.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a4700d640 1 -- 172.21.15.64:0/801706442 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a401e3380 con 0x7f2a400b0510 2024-03-25T18:11:51.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.953+0000 7f2a4700d640 1 librados: init done 2024-03-25T18:11:51.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.954+0000 7f2a4700d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:51.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.954+0000 7f2a4700d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:51.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.954+0000 7f2a4700d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:11:51.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:51.954+0000 7f2a4700d640 1 -- 172.21.15.64:0/801706442 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a400627b0 con 0x7f2a400b0510 2024-03-25T18:11:52.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.275+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a2c0011f0 con 0x7f2a400b0510 2024-03-25T18:11:52.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.275+0000 7f2a3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:52.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.275+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2a2c004a90 con 0x7f2a400b0510 2024-03-25T18:11:52.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.275+0000 7f2a3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:52.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.275+0000 7f2a3d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:52.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.275+0000 7f2a3d7fa640 10 monclient: dump: 2024-03-25T18:11:52.277 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:52.277 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:52.277 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:52.277 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:52.278 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:52.278 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:52.278 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:52.278 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:52.278 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:52.278 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:52.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.275+0000 7f2a3effd640 4 set_mon_vals no callback set 2024-03-25T18:11:52.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.276+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f2a2c01aaa0 con 0x7f2a400b0510 2024-03-25T18:11:52.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.276+0000 7f2a44da9640 10 monclient: get_auth_request con 0x7f2a2406e0d0 auth_method 0 2024-03-25T18:11:52.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.277+0000 7f2a3d7fa640 10 client.4328.objecter ms_handle_connect 0x7f2a2406e0d0 2024-03-25T18:11:52.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.283+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f2a2c0547a0 con 0x7f2a400b0510 2024-03-25T18:11:52.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.283+0000 7f2a3d7fa640 10 client.4328.objecter ms_dispatch 0x7f2a40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:52.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.283+0000 7f2a3d7fa640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:52.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.283+0000 7f2a3d7fa640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:52.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.283+0000 7f2a3d7fa640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-25T18:11:52.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.283+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a2c0500e0 con 0x7f2a400b0510 2024-03-25T18:11:52.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.283+0000 7f2a3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:52.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.283+0000 7f2a3d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:52.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.537+0000 7f2a4700d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:52.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.537+0000 7f2a4700d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:52.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.537+0000 7f2a4700d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:11:52.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.537+0000 7f2a4700d640 1 -- 172.21.15.64:0/801706442 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a4013d0c0 con 0x7f2a400b0510 2024-03-25T18:11:52.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.585+0000 7f2a3d7fa640 1 -- 172.21.15.64:0/801706442 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f2a2c0521e0 con 0x7f2a400b0510 2024-03-25T18:11:52.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.585+0000 7f2a3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:52.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.585+0000 7f2a3d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:52.587 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:11:52.587 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:52.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.588+0000 7f2a4700d640 10 librados: watch_flush enter 2024-03-25T18:11:52.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.588+0000 7f2a4700d640 10 librados: watch_flush exit 2024-03-25T18:11:52.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.588+0000 7f2a4700d640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-25T18:11:52.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.588+0000 7f2a4700d640 10 client.4328.objecter successfully canceled tick 2024-03-25T18:11:52.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.588+0000 7f2a4700d640 1 -- 172.21.15.64:0/801706442 >> v1:172.21.15.64:6800/2915543638 conn(0x7f2a2406e0d0 legacy=0x7f2a24070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:52.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.588+0000 7f2a4700d640 10 monclient: shutdown 2024-03-25T18:11:52.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.588+0000 7f2a4700d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:52.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.588+0000 7f2a4700d640 1 -- 172.21.15.64:0/801706442 >> v1:172.21.15.64:6790/0 conn(0x7f2a400b0510 legacy=0x7f2a4011eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:52.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.588+0000 7f2a4700d640 1 -- 172.21.15.64:0/801706442 shutdown_connections 2024-03-25T18:11:52.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.588+0000 7f2a4700d640 1 -- 172.21.15.64:0/801706442 >> 172.21.15.64:0/801706442 conn(0x7f2a400b0ef0 msgr2=0x7f2a400afc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:52.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.589+0000 7f2a4700d640 1 -- 172.21.15.64:0/801706442 shutdown_connections 2024-03-25T18:11:52.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.589+0000 7f2a4700d640 1 -- 172.21.15.64:0/801706442 wait complete. 2024-03-25T18:11:52.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:52.589+0000 7f2a4700d640 1 librados: shutdown 2024-03-25T18:11:52.602 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:59.409 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:59.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.528+0000 7fbbf9a88640 10 monclient: get_monmap_and_config 2024-03-25T18:11:59.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.528+0000 7fbbf9a88640 10 monclient: build_initial_monmap 2024-03-25T18:11:59.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.528+0000 7fbbf9a88640 10 monclient: monmap: 2024-03-25T18:11:59.529 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:11:59.529 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:59.529 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:11:59.529334+0000 2024-03-25T18:11:59.529 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:11:59.529334+0000 2024-03-25T18:11:59.529 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:59.529 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:59.529 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:11:59.529 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:11:59.530 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:11:59.530 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:59.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.529+0000 7fbbf9a88640 1 Processor -- start 2024-03-25T18:11:59.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.529+0000 7fbbf9a88640 1 -- start start 2024-03-25T18:11:59.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.529+0000 7fbbf9a88640 10 monclient: init 2024-03-25T18:11:59.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.529+0000 7fbbf9a88640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:59.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.529+0000 7fbbf9a88640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:11:59.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.529+0000 7fbbf9a88640 10 monclient(hunting): picked mon.noname-b con 0x7fbbf413fc60 addr v1:172.21.15.64:6790/0 2024-03-25T18:11:59.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.529+0000 7fbbf9a88640 10 monclient(hunting): picked mon.noname-a con 0x7fbbf4070570 addr v1:172.21.15.64:6789/0 2024-03-25T18:11:59.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.530+0000 7fbbf9a88640 10 monclient(hunting): picked mon.noname-c con 0x7fbbf40727a0 addr v1:172.21.15.86:6789/0 2024-03-25T18:11:59.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.530+0000 7fbbf9a88640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf413c9f0 con 0x7fbbf4070570 2024-03-25T18:11:59.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.530+0000 7fbbf9a88640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf406fac0 con 0x7fbbf40727a0 2024-03-25T18:11:59.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.530+0000 7fbbf9a88640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf40715d0 con 0x7fbbf413fc60 2024-03-25T18:11:59.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.530+0000 7fbbf9a88640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:59.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.530+0000 7fbbf9a88640 10 monclient(hunting): authenticate will time out at 7105.768550s 2024-03-25T18:11:59.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.544+0000 7fbbf37fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fbbf413fc60 0x7fbbf406f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:45912/0 (socket says 172.21.15.64:45912) 2024-03-25T18:11:59.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.544+0000 7fbbf37fe640 1 -- 172.21.15.64:0/532613025 learned_addr learned my addr 172.21.15.64:0/532613025 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:11:59.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.553+0000 7fbbf27fc640 1 -- 172.21.15.64:0/532613025 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220098817 0 0) 0x7fbbf406fac0 con 0x7fbbf40727a0 2024-03-25T18:11:59.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.553+0000 7fbbf27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:59.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.553+0000 7fbbf27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:59.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.553+0000 7fbbf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:59.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.553+0000 7fbbf27fc640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:11:59.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.553+0000 7fbbf27fc640 1 -- 172.21.15.64:0/532613025 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbdc003520 con 0x7fbbf40727a0 2024-03-25T18:11:59.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.560+0000 7fbbf27fc640 1 -- 172.21.15.64:0/532613025 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662616615 0 0) 0x7fbbf40715d0 con 0x7fbbf413fc60 2024-03-25T18:11:59.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.560+0000 7fbbf27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:59.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.560+0000 7fbbf27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:59.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.560+0000 7fbbf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:59.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.560+0000 7fbbf27fc640 10 monclient(hunting): my global_id is 4331 2024-03-25T18:11:59.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.560+0000 7fbbf27fc640 1 -- 172.21.15.64:0/532613025 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf406fac0 con 0x7fbbf413fc60 2024-03-25T18:11:59.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.569+0000 7fbbf27fc640 1 -- 172.21.15.64:0/532613025 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83305743 0 0) 0x7fbbf413c9f0 con 0x7fbbf4070570 2024-03-25T18:11:59.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.569+0000 7fbbf27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:59.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.569+0000 7fbbf27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:59.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.569+0000 7fbbf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:59.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.569+0000 7fbbf27fc640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:11:59.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.569+0000 7fbbf27fc640 1 -- 172.21.15.64:0/532613025 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf40715d0 con 0x7fbbf4070570 2024-03-25T18:11:59.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.577+0000 7fbbf27fc640 1 -- 172.21.15.64:0/532613025 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 207488210 0 0) 0x7fbbdc003520 con 0x7fbbf40727a0 2024-03-25T18:11:59.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.577+0000 7fbbf27fc640 1 -- 172.21.15.64:0/532613025 >> v1:172.21.15.64:6790/0 conn(0x7fbbf413fc60 legacy=0x7fbbf406f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:59.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.577+0000 7fbbf27fc640 1 -- 172.21.15.64:0/532613025 >> v1:172.21.15.64:6789/0 conn(0x7fbbf4070570 legacy=0x7fbbf4072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:59.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.577+0000 7fbbf27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:11:59.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.577+0000 7fbbf27fc640 1 monclient: found mon.noname-c 2024-03-25T18:11:59.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.577+0000 7fbbf27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:59.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.577+0000 7fbbf27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:11:59.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.577+0000 7fbbf27fc640 1 -- 172.21.15.64:0/532613025 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf40718b0 con 0x7fbbf40727a0 2024-03-25T18:11:59.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.577+0000 7fbbf27fc640 10 monclient: _finish_auth 0 2024-03-25T18:11:59.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.577+0000 7fbbf27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:11:59.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.577+0000 7fbbf27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:59.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.577+0000 7fbbf27fc640 1 -- 172.21.15.64:0/532613025 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbbec001a10 con 0x7fbbf40727a0 2024-03-25T18:11:59.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.577+0000 7fbbf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:59.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.577+0000 7fbbf27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:59.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.577+0000 7fbbf27fc640 10 monclient: dump: 2024-03-25T18:11:59.579 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:59.579 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:59.579 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:59.579 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:59.579 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:59.579 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:59.579 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:59.579 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:59.579 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:59.579 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:59.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.577+0000 7fbbf9a88640 5 monclient: authenticate success, global_id 4321 2024-03-25T18:11:59.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.577+0000 7fbbf9a88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:59.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.587+0000 7fbbf27fc640 1 -- 172.21.15.64:0/532613025 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbec001b70 con 0x7fbbf40727a0 2024-03-25T18:11:59.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.587+0000 7fbbf27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:59.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.587+0000 7fbbf27fc640 1 -- 172.21.15.64:0/532613025 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbbec001f80 con 0x7fbbf40727a0 2024-03-25T18:11:59.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.587+0000 7fbbf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:59.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.587+0000 7fbbf27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:59.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.587+0000 7fbbf27fc640 10 monclient: dump: 2024-03-25T18:11:59.588 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:59.588 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:59.589 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:59.589 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:59.589 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:59.589 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:59.589 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:59.589 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:59.589 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:59.589 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:59.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.587+0000 7fbbf9a88640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:59.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.587+0000 7fbbf9a88640 4 set_mon_vals no callback set 2024-03-25T18:11:59.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.588+0000 7fbbf9a88640 10 monclient: shutdown 2024-03-25T18:11:59.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.588+0000 7fbbf9a88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:59.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.588+0000 7fbbf9a88640 1 -- 172.21.15.64:0/532613025 >> v1:172.21.15.86:6789/0 conn(0x7fbbf40727a0 legacy=0x7fbbf414a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:59.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.588+0000 7fbbf9a88640 1 -- 172.21.15.64:0/532613025 shutdown_connections 2024-03-25T18:11:59.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.588+0000 7fbbf9a88640 1 -- 172.21.15.64:0/532613025 >> 172.21.15.64:0/532613025 conn(0x7fbbf40b4a80 msgr2=0x7fbbf40b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:59.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.588+0000 7fbbf9a88640 1 -- 172.21.15.64:0/532613025 shutdown_connections 2024-03-25T18:11:59.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.588+0000 7fbbf9a88640 1 -- 172.21.15.64:0/532613025 wait complete. 2024-03-25T18:11:59.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.589+0000 7fbbf9a88640 10 monclient: build_initial_monmap 2024-03-25T18:11:59.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.589+0000 7fbbf9a88640 10 monclient: monmap: 2024-03-25T18:11:59.592 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:11:59.592 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:59.592 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:11:59.592 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:11:59.592 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:59.592 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:11:59.592 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:11:59.592 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:11:59.592 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:11:59.592 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:59.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.589+0000 7fbbf9a88640 1 librados: starting msgr at 2024-03-25T18:11:59.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.589+0000 7fbbf9a88640 1 librados: starting objecter 2024-03-25T18:11:59.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.589+0000 7fbbf9a88640 1 Processor -- start 2024-03-25T18:11:59.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.590+0000 7fbbf9a88640 1 -- start start 2024-03-25T18:11:59.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.590+0000 7fbbf9a88640 1 librados: setting wanted keys 2024-03-25T18:11:59.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.590+0000 7fbbf9a88640 1 librados: calling monclient init 2024-03-25T18:11:59.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.590+0000 7fbbf9a88640 10 monclient: init 2024-03-25T18:11:59.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.590+0000 7fbbf9a88640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:59.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.590+0000 7fbbf9a88640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:11:59.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.590+0000 7fbbf9a88640 10 monclient(hunting): picked mon.noname-c con 0x7fbbf4070570 addr v1:172.21.15.64:6790/0 2024-03-25T18:11:59.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.590+0000 7fbbf9a88640 10 monclient(hunting): picked mon.noname-b con 0x7fbbf40727a0 addr v1:172.21.15.86:6789/0 2024-03-25T18:11:59.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.590+0000 7fbbf9a88640 10 monclient(hunting): picked mon.noname-a con 0x7fbbf413fc60 addr v1:172.21.15.64:6789/0 2024-03-25T18:11:59.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.590+0000 7fbbf9a88640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf4133a10 con 0x7fbbf413fc60 2024-03-25T18:11:59.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.590+0000 7fbbf9a88640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf4133d90 con 0x7fbbf40727a0 2024-03-25T18:11:59.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.590+0000 7fbbf9a88640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf41419a0 con 0x7fbbf4070570 2024-03-25T18:11:59.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.590+0000 7fbbf9a88640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:59.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.590+0000 7fbbf9a88640 10 monclient(hunting): authenticate will time out at 7105.829350s 2024-03-25T18:11:59.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.605+0000 7fbbf3fff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fbbf413fc60 0x7fbbf4133300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:33766/0 (socket says 172.21.15.64:33766) 2024-03-25T18:11:59.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.605+0000 7fbbf3fff640 1 -- 172.21.15.64:0/3349963654 learned_addr learned my addr 172.21.15.64:0/3349963654 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:11:59.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.607+0000 7fbbdbfff640 10 client.?.objecter ms_handle_connect 0x7fbbf413fc60 2024-03-25T18:11:59.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.607+0000 7fbbdbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:59.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.608+0000 7fbbdbfff640 10 client.?.objecter ms_handle_connect 0x7fbbf40727a0 2024-03-25T18:11:59.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.608+0000 7fbbdbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:59.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.610+0000 7fbbdbfff640 10 client.?.objecter ms_handle_connect 0x7fbbf4070570 2024-03-25T18:11:59.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.610+0000 7fbbdbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:59.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.617+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338202918 0 0) 0x7fbbf4133d90 con 0x7fbbf40727a0 2024-03-25T18:11:59.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.617+0000 7fbbdbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:59.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.617+0000 7fbbdbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:59.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.617+0000 7fbbdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:59.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.617+0000 7fbbdbfff640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:11:59.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.617+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbe0004eb0 con 0x7fbbf40727a0 2024-03-25T18:11:59.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.623+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581923835 0 0) 0x7fbbf4133a10 con 0x7fbbf413fc60 2024-03-25T18:11:59.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.623+0000 7fbbdbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:59.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.623+0000 7fbbdbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:59.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.623+0000 7fbbdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:59.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.623+0000 7fbbdbfff640 10 monclient(hunting): my global_id is 4332 2024-03-25T18:11:59.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.623+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf4133d90 con 0x7fbbf413fc60 2024-03-25T18:11:59.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.626+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930621301 0 0) 0x7fbbf41419a0 con 0x7fbbf4070570 2024-03-25T18:11:59.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.626+0000 7fbbdbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:59.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.626+0000 7fbbdbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:59.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.626+0000 7fbbdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:59.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.626+0000 7fbbdbfff640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:11:59.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.626+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf4133a10 con 0x7fbbf4070570 2024-03-25T18:11:59.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.641+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1760896624 0 0) 0x7fbbe0004eb0 con 0x7fbbf40727a0 2024-03-25T18:11:59.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.641+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbf41419a0 con 0x7fbbf40727a0 2024-03-25T18:11:59.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.641+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbbe80018f0 con 0x7fbbf40727a0 2024-03-25T18:11:59.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.641+0000 7fbbdbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:59.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.641+0000 7fbbdbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:59.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.641+0000 7fbbdbfff640 10 monclient(hunting): dump: 2024-03-25T18:11:59.642 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:59.642 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:59.642 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:59.642 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:59.642 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:59.642 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:59.643 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:59.643 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:59.643 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:59.643 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:59.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.641+0000 7fbbdbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:59.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.649+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1325326347 0 0) 0x7fbbf4133d90 con 0x7fbbf413fc60 2024-03-25T18:11:59.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.649+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbe0004eb0 con 0x7fbbf413fc60 2024-03-25T18:11:59.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.649+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbbec001e80 con 0x7fbbf413fc60 2024-03-25T18:11:59.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.649+0000 7fbbdbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:59.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.649+0000 7fbbdbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:59.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.649+0000 7fbbdbfff640 10 monclient(hunting): dump: 2024-03-25T18:11:59.650 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:59.650 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:59.650 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:59.650 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:59.650 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:59.650 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:59.650 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:59.650 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:59.650 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:59.650 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:59.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4109817299 0 0) 0x7fbbf41419a0 con 0x7fbbf40727a0 2024-03-25T18:11:59.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 >> v1:172.21.15.64:6790/0 conn(0x7fbbf4070570 legacy=0x7fbbf4142510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:59.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 >> v1:172.21.15.64:6789/0 conn(0x7fbbf413fc60 legacy=0x7fbbf4133300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:59.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbdbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:11:59.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbdbfff640 1 monclient: found mon.b 2024-03-25T18:11:59.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbdbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:59.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbdbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:11:59.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf4141ff0 con 0x7fbbf40727a0 2024-03-25T18:11:59.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbdbfff640 10 monclient: _finish_auth 0 2024-03-25T18:11:59.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbdbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:11:59.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:59.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbf9a88640 5 monclient: authenticate success, global_id 4324 2024-03-25T18:11:59.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbf9a88640 10 monclient: _renew_subs 2024-03-25T18:11:59.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbf9a88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:11:59.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbf9a88640 1 -- 172.21.15.64:0/3349963654 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf41431e0 con 0x7fbbf40727a0 2024-03-25T18:11:59.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbf9a88640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:59.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbf9a88640 10 monclient: _renew_subs 2024-03-25T18:11:59.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbf9a88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:11:59.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbf9a88640 1 -- 172.21.15.64:0/3349963654 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf4136cf0 con 0x7fbbf40727a0 2024-03-25T18:11:59.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.660+0000 7fbbf9a88640 1 librados: init done 2024-03-25T18:11:59.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.661+0000 7fbbf9a88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:59.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.661+0000 7fbbf9a88640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:59.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.661+0000 7fbbf9a88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:11:59.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.661+0000 7fbbf9a88640 1 -- 172.21.15.64:0/3349963654 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf4142230 con 0x7fbbf40727a0 2024-03-25T18:11:59.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.941+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbe8003610 con 0x7fbbf40727a0 2024-03-25T18:11:59.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.941+0000 7fbbdbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:59.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.941+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbbe8003780 con 0x7fbbf40727a0 2024-03-25T18:11:59.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.941+0000 7fbbdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:59.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.941+0000 7fbbdbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:59.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.941+0000 7fbbdbfff640 10 monclient: dump: 2024-03-25T18:11:59.943 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:11:59.943 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:11:59.943 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:59.943 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:11:59.943 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:59.943 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:11:59.943 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:11:59.943 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:11:59.944 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:11:59.944 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:11:59.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.941+0000 7fbbf1ffb640 4 set_mon_vals no callback set 2024-03-25T18:11:59.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.942+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fbbe8019790 con 0x7fbbf40727a0 2024-03-25T18:11:59.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.943+0000 7fbbf37fe640 10 monclient: get_auth_request con 0x7fbbe006df70 auth_method 0 2024-03-25T18:11:59.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.943+0000 7fbbdbfff640 10 client.4324.objecter ms_handle_connect 0x7fbbe006df70 2024-03-25T18:11:59.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.948+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fbbe80534b0 con 0x7fbbf40727a0 2024-03-25T18:11:59.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.948+0000 7fbbdbfff640 10 client.4324.objecter ms_dispatch 0x7fbbf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:59.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.948+0000 7fbbdbfff640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:59.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.948+0000 7fbbdbfff640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:59.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.948+0000 7fbbdbfff640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-25T18:11:59.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.948+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbbe804edf0 con 0x7fbbf40727a0 2024-03-25T18:11:59.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.948+0000 7fbbdbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:59.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:11:59.948+0000 7fbbdbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:00.183 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.182+0000 7fbbf9a88640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:00.183 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.182+0000 7fbbf9a88640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:00.183 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.182+0000 7fbbf9a88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:12:00.183 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.182+0000 7fbbf9a88640 1 -- 172.21.15.64:0/3349963654 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbf4143370 con 0x7fbbf40727a0 2024-03-25T18:12:00.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.212+0000 7fbbdbfff640 1 -- 172.21.15.64:0/3349963654 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fbbe8050ef0 con 0x7fbbf40727a0 2024-03-25T18:12:00.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.212+0000 7fbbdbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:00.213 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.212+0000 7fbbdbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:00.213 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:12:00.213 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:00.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.214+0000 7fbbf9a88640 10 librados: watch_flush enter 2024-03-25T18:12:00.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.214+0000 7fbbf9a88640 10 librados: watch_flush exit 2024-03-25T18:12:00.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.214+0000 7fbbf9a88640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-25T18:12:00.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.214+0000 7fbbf9a88640 10 client.4324.objecter successfully canceled tick 2024-03-25T18:12:00.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.214+0000 7fbbf9a88640 1 -- 172.21.15.64:0/3349963654 >> v1:172.21.15.64:6800/2915543638 conn(0x7fbbe006df70 legacy=0x7fbbe0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:00.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.214+0000 7fbbf9a88640 10 monclient: shutdown 2024-03-25T18:12:00.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.214+0000 7fbbf9a88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:00.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.214+0000 7fbbf9a88640 1 -- 172.21.15.64:0/3349963654 >> v1:172.21.15.86:6789/0 conn(0x7fbbf40727a0 legacy=0x7fbbf4132bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:00.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.214+0000 7fbbf9a88640 1 -- 172.21.15.64:0/3349963654 shutdown_connections 2024-03-25T18:12:00.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.214+0000 7fbbf9a88640 1 -- 172.21.15.64:0/3349963654 >> 172.21.15.64:0/3349963654 conn(0x7fbbf40b4a80 msgr2=0x7fbbf406b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:00.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.215+0000 7fbbf9a88640 1 -- 172.21.15.64:0/3349963654 shutdown_connections 2024-03-25T18:12:00.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.215+0000 7fbbf9a88640 1 -- 172.21.15.64:0/3349963654 wait complete. 2024-03-25T18:12:00.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:00.215+0000 7fbbf9a88640 1 librados: shutdown 2024-03-25T18:12:00.228 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:07.033 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:07.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.145+0000 7fccc53ab640 10 monclient: get_monmap_and_config 2024-03-25T18:12:07.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.145+0000 7fccc53ab640 10 monclient: build_initial_monmap 2024-03-25T18:12:07.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.145+0000 7fccc53ab640 10 monclient: monmap: 2024-03-25T18:12:07.145 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:12:07.146 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:07.146 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:12:07.145919+0000 2024-03-25T18:12:07.146 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:12:07.145919+0000 2024-03-25T18:12:07.146 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:07.146 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:07.146 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:12:07.146 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:12:07.146 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:12:07.146 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:07.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.145+0000 7fccc53ab640 1 Processor -- start 2024-03-25T18:12:07.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.145+0000 7fccc53ab640 1 -- start start 2024-03-25T18:12:07.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.145+0000 7fccc53ab640 10 monclient: init 2024-03-25T18:12:07.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.146+0000 7fccc53ab640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:07.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.146+0000 7fccc53ab640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:07.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.146+0000 7fccc53ab640 10 monclient(hunting): picked mon.noname-a con 0x7fccc00a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:12:07.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.146+0000 7fccc53ab640 10 monclient(hunting): picked mon.noname-b con 0x7fccc013f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:12:07.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.146+0000 7fccc53ab640 10 monclient(hunting): picked mon.noname-c con 0x7fccc01430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:12:07.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.146+0000 7fccc53ab640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc0142520 con 0x7fccc00a98f0 2024-03-25T18:12:07.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.146+0000 7fccc53ab640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc00b28a0 con 0x7fccc01430c0 2024-03-25T18:12:07.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.146+0000 7fccc53ab640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc00b2a30 con 0x7fccc013f710 2024-03-25T18:12:07.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.146+0000 7fccc53ab640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:07.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.146+0000 7fccc53ab640 10 monclient(hunting): authenticate will time out at 7113.385263s 2024-03-25T18:12:07.150 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.150+0000 7fccbe7fc640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fccc013f710 0x7fccc01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:38094/0 (socket says 172.21.15.64:38094) 2024-03-25T18:12:07.151 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.150+0000 7fccbe7fc640 1 -- 172.21.15.64:0/1889374535 learned_addr learned my addr 172.21.15.64:0/1889374535 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:12:07.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.163+0000 7fccbdffb640 1 -- 172.21.15.64:0/1889374535 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554564629 0 0) 0x7fccc00b2a30 con 0x7fccc013f710 2024-03-25T18:12:07.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.163+0000 7fccbdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:07.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.163+0000 7fccbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:07.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.163+0000 7fccbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:07.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.163+0000 7fccbdffb640 10 monclient(hunting): my global_id is 4337 2024-03-25T18:12:07.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.163+0000 7fccbdffb640 1 -- 172.21.15.64:0/1889374535 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca8003520 con 0x7fccc013f710 2024-03-25T18:12:07.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.169+0000 7fccbdffb640 1 -- 172.21.15.64:0/1889374535 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525225344 0 0) 0x7fccc0142520 con 0x7fccc00a98f0 2024-03-25T18:12:07.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.169+0000 7fccbdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:07.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.169+0000 7fccbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:07.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.169+0000 7fccbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:07.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.169+0000 7fccbdffb640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:12:07.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.169+0000 7fccbdffb640 1 -- 172.21.15.64:0/1889374535 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc00b2a30 con 0x7fccc00a98f0 2024-03-25T18:12:07.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.174+0000 7fccbdffb640 1 -- 172.21.15.64:0/1889374535 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069395126 0 0) 0x7fccc00b28a0 con 0x7fccc01430c0 2024-03-25T18:12:07.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.174+0000 7fccbdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:07.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.174+0000 7fccbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:07.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.174+0000 7fccbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:07.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.174+0000 7fccbdffb640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:12:07.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.174+0000 7fccbdffb640 1 -- 172.21.15.64:0/1889374535 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc0142520 con 0x7fccc01430c0 2024-03-25T18:12:07.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.178+0000 7fccbdffb640 1 -- 172.21.15.64:0/1889374535 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1014763226 0 0) 0x7fcca8003520 con 0x7fccc013f710 2024-03-25T18:12:07.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.178+0000 7fccbdffb640 1 -- 172.21.15.64:0/1889374535 >> v1:172.21.15.86:6789/0 conn(0x7fccc01430c0 legacy=0x7fccc014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:07.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.178+0000 7fccbdffb640 1 -- 172.21.15.64:0/1889374535 >> v1:172.21.15.64:6789/0 conn(0x7fccc00a98f0 legacy=0x7fccc00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:07.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.178+0000 7fccbdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:12:07.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.178+0000 7fccbdffb640 1 monclient: found mon.noname-b 2024-03-25T18:12:07.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.178+0000 7fccbdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:07.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.178+0000 7fccbdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:12:07.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.178+0000 7fccbdffb640 1 -- 172.21.15.64:0/1889374535 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc00b44c0 con 0x7fccc013f710 2024-03-25T18:12:07.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.178+0000 7fccbdffb640 10 monclient: _finish_auth 0 2024-03-25T18:12:07.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.178+0000 7fccbdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:12:07.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.178+0000 7fccbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:07.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.178+0000 7fccbdffb640 1 -- 172.21.15.64:0/1889374535 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fccb4001690 con 0x7fccc013f710 2024-03-25T18:12:07.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.178+0000 7fccbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:07.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.178+0000 7fccbdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:07.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.178+0000 7fccbdffb640 10 monclient: dump: 2024-03-25T18:12:07.180 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:07.180 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:07.180 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:07.180 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:07.180 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:07.181 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:07.181 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:07.181 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:07.181 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:07.181 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:07.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.178+0000 7fccc53ab640 5 monclient: authenticate success, global_id 4337 2024-03-25T18:12:07.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.178+0000 7fccc53ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:07.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.197+0000 7fccbdffb640 1 -- 172.21.15.64:0/1889374535 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccb40017f0 con 0x7fccc013f710 2024-03-25T18:12:07.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.197+0000 7fccbdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:07.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.197+0000 7fccbdffb640 1 -- 172.21.15.64:0/1889374535 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fccb4001c00 con 0x7fccc013f710 2024-03-25T18:12:07.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.197+0000 7fccbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:07.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.197+0000 7fccbdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:07.198 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.197+0000 7fccbdffb640 10 monclient: dump: 2024-03-25T18:12:07.198 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:07.198 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:07.198 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:07.198 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:07.198 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:07.199 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:07.199 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:07.199 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:07.199 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:07.199 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:07.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.197+0000 7fccc53ab640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:07.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.197+0000 7fccc53ab640 4 set_mon_vals no callback set 2024-03-25T18:12:07.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.198+0000 7fccc53ab640 10 monclient: shutdown 2024-03-25T18:12:07.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.198+0000 7fccc53ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:07.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.198+0000 7fccc53ab640 1 -- 172.21.15.64:0/1889374535 >> v1:172.21.15.64:6790/0 conn(0x7fccc013f710 legacy=0x7fccc01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:07.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.198+0000 7fccc53ab640 1 -- 172.21.15.64:0/1889374535 shutdown_connections 2024-03-25T18:12:07.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.198+0000 7fccc53ab640 1 -- 172.21.15.64:0/1889374535 >> 172.21.15.64:0/1889374535 conn(0x7fccc00b7ea0 msgr2=0x7fccc00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:07.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.198+0000 7fccc53ab640 1 -- 172.21.15.64:0/1889374535 shutdown_connections 2024-03-25T18:12:07.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.198+0000 7fccc53ab640 1 -- 172.21.15.64:0/1889374535 wait complete. 2024-03-25T18:12:07.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.199+0000 7fccc53ab640 10 monclient: build_initial_monmap 2024-03-25T18:12:07.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.199+0000 7fccc53ab640 10 monclient: monmap: 2024-03-25T18:12:07.202 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:12:07.202 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:07.202 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:12:07.202 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:12:07.202 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:07.202 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:12:07.202 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:12:07.202 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:12:07.202 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:12:07.202 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:07.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.199+0000 7fccc53ab640 1 librados: starting msgr at 2024-03-25T18:12:07.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.199+0000 7fccc53ab640 1 librados: starting objecter 2024-03-25T18:12:07.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.199+0000 7fccc53ab640 1 Processor -- start 2024-03-25T18:12:07.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.199+0000 7fccc53ab640 1 -- start start 2024-03-25T18:12:07.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.199+0000 7fccc53ab640 1 librados: setting wanted keys 2024-03-25T18:12:07.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.199+0000 7fccc53ab640 1 librados: calling monclient init 2024-03-25T18:12:07.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.199+0000 7fccc53ab640 10 monclient: init 2024-03-25T18:12:07.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.200+0000 7fccc53ab640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:07.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.200+0000 7fccc53ab640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:07.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.200+0000 7fccc53ab640 10 monclient(hunting): picked mon.noname-b con 0x7fccc00a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:12:07.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.200+0000 7fccc53ab640 10 monclient(hunting): picked mon.noname-a con 0x7fccc013f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:12:07.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.200+0000 7fccc53ab640 10 monclient(hunting): picked mon.noname-c con 0x7fccc01430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:12:07.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.200+0000 7fccc53ab640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc00b1870 con 0x7fccc013f710 2024-03-25T18:12:07.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.200+0000 7fccc53ab640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc00ac910 con 0x7fccc00a98f0 2024-03-25T18:12:07.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.200+0000 7fccc53ab640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc00abb00 con 0x7fccc01430c0 2024-03-25T18:12:07.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.200+0000 7fccc53ab640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:07.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.200+0000 7fccc53ab640 10 monclient(hunting): authenticate will time out at 7113.439014s 2024-03-25T18:12:07.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.211+0000 7fccbeffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fccc00a98f0 0x7fccc0138670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:42980/0 (socket says 172.21.15.64:42980) 2024-03-25T18:12:07.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.211+0000 7fccbeffd640 1 -- 172.21.15.64:0/2126246116 learned_addr learned my addr 172.21.15.64:0/2126246116 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:12:07.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.212+0000 7fcca77fe640 10 client.?.objecter ms_handle_connect 0x7fccc00a98f0 2024-03-25T18:12:07.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.212+0000 7fcca77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:07.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.219+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277083859 0 0) 0x7fccc00ac910 con 0x7fccc00a98f0 2024-03-25T18:12:07.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.219+0000 7fcca77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:07.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.219+0000 7fcca77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:07.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.219+0000 7fcca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:07.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.219+0000 7fcca77fe640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:12:07.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.219+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccac004eb0 con 0x7fccc00a98f0 2024-03-25T18:12:07.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.220+0000 7fcca77fe640 10 client.?.objecter ms_handle_connect 0x7fccc013f710 2024-03-25T18:12:07.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.220+0000 7fcca77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:07.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.222+0000 7fcca77fe640 10 client.?.objecter ms_handle_connect 0x7fccc01430c0 2024-03-25T18:12:07.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.222+0000 7fcca77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:07.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.231+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025442523 0 0) 0x7fccc00abb00 con 0x7fccc01430c0 2024-03-25T18:12:07.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.231+0000 7fcca77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:07.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.231+0000 7fcca77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:07.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.231+0000 7fcca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:07.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.231+0000 7fcca77fe640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:12:07.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.231+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc00ac910 con 0x7fccc01430c0 2024-03-25T18:12:07.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.243+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1350581348 0 0) 0x7fccc00ac910 con 0x7fccc01430c0 2024-03-25T18:12:07.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.243+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccc00abb00 con 0x7fccc01430c0 2024-03-25T18:12:07.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.245+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2757778405 0 0) 0x7fccac004eb0 con 0x7fccc00a98f0 2024-03-25T18:12:07.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.245+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccc00ac910 con 0x7fccc00a98f0 2024-03-25T18:12:07.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.245+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fccb0001a30 con 0x7fccc00a98f0 2024-03-25T18:12:07.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.245+0000 7fcca77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:07.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.245+0000 7fcca77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:07.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.245+0000 7fcca77fe640 10 monclient(hunting): dump: 2024-03-25T18:12:07.246 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:07.246 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:07.246 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:07.246 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:07.246 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:07.246 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:07.247 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:07.247 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:07.247 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:07.247 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:07.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.245+0000 7fcca77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:07.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.245+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145391835 0 0) 0x7fccc00b44c0 con 0x7fccc013f710 2024-03-25T18:12:07.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.245+0000 7fcca77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:07.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.245+0000 7fcca77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:07.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.245+0000 7fcca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:07.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.245+0000 7fcca77fe640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:12:07.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.245+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccac004eb0 con 0x7fccc013f710 2024-03-25T18:12:07.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.245+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fccb8001b90 con 0x7fccc01430c0 2024-03-25T18:12:07.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.245+0000 7fcca77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:07.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.245+0000 7fcca77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:07.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.245+0000 7fcca77fe640 10 monclient(hunting): dump: 2024-03-25T18:12:07.247 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:07.247 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:07.248 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:07.248 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:07.248 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:07.248 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:07.248 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:07.248 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:07.248 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:07.248 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:07.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 918869974 0 0) 0x7fccc00abb00 con 0x7fccc01430c0 2024-03-25T18:12:07.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 >> v1:172.21.15.86:6789/0 conn(0x7fccc00a98f0 legacy=0x7fccc0138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:07.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 >> v1:172.21.15.64:6789/0 conn(0x7fccc013f710 legacy=0x7fccc00ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:07.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fcca77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:12:07.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fcca77fe640 1 monclient: found mon.c 2024-03-25T18:12:07.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fcca77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:07.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fcca77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:12:07.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc00abde0 con 0x7fccc01430c0 2024-03-25T18:12:07.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fcca77fe640 10 monclient: _finish_auth 0 2024-03-25T18:12:07.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fcca77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:12:07.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fcca77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:07.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fccc53ab640 5 monclient: authenticate success, global_id 4340 2024-03-25T18:12:07.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fccc53ab640 10 monclient: _renew_subs 2024-03-25T18:12:07.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fccc53ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:12:07.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fccc53ab640 1 -- 172.21.15.64:0/2126246116 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc0138f40 con 0x7fccc01430c0 2024-03-25T18:12:07.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fccc53ab640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:07.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fccc53ab640 10 monclient: _renew_subs 2024-03-25T18:12:07.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fccc53ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:12:07.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fccc53ab640 1 -- 172.21.15.64:0/2126246116 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc01363f0 con 0x7fccc01430c0 2024-03-25T18:12:07.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.257+0000 7fccc53ab640 1 librados: init done 2024-03-25T18:12:07.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.258+0000 7fccc53ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:07.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.258+0000 7fccc53ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:07.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.258+0000 7fccc53ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:12:07.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.258+0000 7fccc53ab640 1 -- 172.21.15.64:0/2126246116 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc00627b0 con 0x7fccc01430c0 2024-03-25T18:12:07.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.591+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccb8002060 con 0x7fccc01430c0 2024-03-25T18:12:07.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.591+0000 7fcca77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:07.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.591+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fccb8003b30 con 0x7fccc01430c0 2024-03-25T18:12:07.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.591+0000 7fcca77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:07.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.591+0000 7fcca77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:07.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.591+0000 7fcca77fe640 10 monclient: dump: 2024-03-25T18:12:07.593 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:07.593 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:07.593 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:07.593 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:07.593 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:07.593 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:07.593 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:07.593 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:07.593 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:07.593 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:07.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.591+0000 7fccbd7fa640 4 set_mon_vals no callback set 2024-03-25T18:12:07.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.592+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fccb8003d50 con 0x7fccc01430c0 2024-03-25T18:12:07.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.593+0000 7fccbeffd640 10 monclient: get_auth_request con 0x7fccac06df60 auth_method 0 2024-03-25T18:12:07.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.593+0000 7fcca77fe640 10 client.4340.objecter ms_handle_connect 0x7fccac06df60 2024-03-25T18:12:07.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.598+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fccb8053880 con 0x7fccc01430c0 2024-03-25T18:12:07.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.598+0000 7fcca77fe640 10 client.4340.objecter ms_dispatch 0x7fccc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:07.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.598+0000 7fcca77fe640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:07.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.598+0000 7fcca77fe640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:07.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.598+0000 7fcca77fe640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-25T18:12:07.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.598+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fccb804f390 con 0x7fccc01430c0 2024-03-25T18:12:07.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.598+0000 7fcca77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:07.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.598+0000 7fcca77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:07.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.847+0000 7fccc53ab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:07.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.847+0000 7fccc53ab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:07.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.847+0000 7fccc53ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:12:07.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.847+0000 7fccc53ab640 1 -- 172.21.15.64:0/2126246116 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccc01434a0 con 0x7fccc01430c0 2024-03-25T18:12:07.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.881+0000 7fcca77fe640 1 -- 172.21.15.64:0/2126246116 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fccb80512c0 con 0x7fccc01430c0 2024-03-25T18:12:07.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.881+0000 7fcca77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:07.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.881+0000 7fcca77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:07.882 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:12:07.882 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:07.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.883+0000 7fccc53ab640 10 librados: watch_flush enter 2024-03-25T18:12:07.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.884+0000 7fccc53ab640 10 librados: watch_flush exit 2024-03-25T18:12:07.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.884+0000 7fccc53ab640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-25T18:12:07.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.884+0000 7fccc53ab640 10 client.4340.objecter successfully canceled tick 2024-03-25T18:12:07.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.884+0000 7fccc53ab640 1 -- 172.21.15.64:0/2126246116 >> v1:172.21.15.64:6800/2915543638 conn(0x7fccac06df60 legacy=0x7fccac070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:07.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.884+0000 7fccc53ab640 10 monclient: shutdown 2024-03-25T18:12:07.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.884+0000 7fccc53ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:07.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.884+0000 7fccc53ab640 1 -- 172.21.15.64:0/2126246116 >> v1:172.21.15.64:6790/0 conn(0x7fccc01430c0 legacy=0x7fccc01e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:07.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.884+0000 7fccc53ab640 1 -- 172.21.15.64:0/2126246116 shutdown_connections 2024-03-25T18:12:07.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.884+0000 7fccc53ab640 1 -- 172.21.15.64:0/2126246116 >> 172.21.15.64:0/2126246116 conn(0x7fccc00b7ea0 msgr2=0x7fccc0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:07.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.884+0000 7fccc53ab640 1 -- 172.21.15.64:0/2126246116 shutdown_connections 2024-03-25T18:12:07.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.884+0000 7fccc53ab640 1 -- 172.21.15.64:0/2126246116 wait complete. 2024-03-25T18:12:07.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:07.884+0000 7fccc53ab640 1 librados: shutdown 2024-03-25T18:12:07.896 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:11.684 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:02:54.797 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:11.693 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:02:54.806 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:11.734 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:02:54.847 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:11.753 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:02:54.866 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:11.753 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:02:54.832 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:11.759 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:02:54.872 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:11.759 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:02:54.838 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:11.786 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:02:54.865 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:11.792 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:02:54.905 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:14.702 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:14.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.820+0000 7f2e9656d640 10 monclient: get_monmap_and_config 2024-03-25T18:12:14.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.820+0000 7f2e9656d640 10 monclient: build_initial_monmap 2024-03-25T18:12:14.821 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.820+0000 7f2e9656d640 10 monclient: monmap: 2024-03-25T18:12:14.821 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:12:14.821 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:14.821 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:12:14.821572+0000 2024-03-25T18:12:14.821 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:12:14.821572+0000 2024-03-25T18:12:14.821 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:14.821 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:14.822 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:12:14.822 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:12:14.822 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:12:14.822 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:14.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.821+0000 7f2e9656d640 1 Processor -- start 2024-03-25T18:12:14.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.821+0000 7f2e9656d640 1 -- start start 2024-03-25T18:12:14.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.821+0000 7f2e9656d640 10 monclient: init 2024-03-25T18:12:14.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.822+0000 7f2e9656d640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:14.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.822+0000 7f2e9656d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:14.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.822+0000 7f2e9656d640 10 monclient(hunting): picked mon.noname-b con 0x7f2e900a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:12:14.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.822+0000 7f2e9656d640 10 monclient(hunting): picked mon.noname-a con 0x7f2e9013f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:12:14.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.822+0000 7f2e9656d640 10 monclient(hunting): picked mon.noname-c con 0x7f2e901430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:12:14.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.822+0000 7f2e9656d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e90142520 con 0x7f2e9013f710 2024-03-25T18:12:14.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.822+0000 7f2e9656d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e900b28a0 con 0x7f2e901430c0 2024-03-25T18:12:14.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.822+0000 7f2e9656d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e900b2a30 con 0x7f2e900a98f0 2024-03-25T18:12:14.824 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.822+0000 7f2e9656d640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:14.824 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.822+0000 7f2e9656d640 10 monclient(hunting): authenticate will time out at 7121.061061s 2024-03-25T18:12:14.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.832+0000 7f2e94b0a640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2e901430c0 0x7f2e9014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:47416/0 (socket says 172.21.15.64:47416) 2024-03-25T18:12:14.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.832+0000 7f2e94b0a640 1 -- 172.21.15.64:0/2138997376 learned_addr learned my addr 172.21.15.64:0/2138997376 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:12:14.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.837+0000 7f2e8effd640 1 -- 172.21.15.64:0/2138997376 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1804416679 0 0) 0x7f2e900b28a0 con 0x7f2e901430c0 2024-03-25T18:12:14.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.838+0000 7f2e8effd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:14.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.838+0000 7f2e8effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:14.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.838+0000 7f2e8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:14.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.838+0000 7f2e8effd640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:12:14.839 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.838+0000 7f2e8effd640 1 -- 172.21.15.64:0/2138997376 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e78003520 con 0x7f2e901430c0 2024-03-25T18:12:14.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.849+0000 7f2e8effd640 1 -- 172.21.15.64:0/2138997376 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968755537 0 0) 0x7f2e900b2a30 con 0x7f2e900a98f0 2024-03-25T18:12:14.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.849+0000 7f2e8effd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:14.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.849+0000 7f2e8effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:14.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.849+0000 7f2e8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:14.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.849+0000 7f2e8effd640 10 monclient(hunting): my global_id is 4343 2024-03-25T18:12:14.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.849+0000 7f2e8effd640 1 -- 172.21.15.64:0/2138997376 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e900b28a0 con 0x7f2e900a98f0 2024-03-25T18:12:14.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.855+0000 7f2e8effd640 1 -- 172.21.15.64:0/2138997376 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2581689587 0 0) 0x7f2e78003520 con 0x7f2e901430c0 2024-03-25T18:12:14.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.855+0000 7f2e8effd640 1 -- 172.21.15.64:0/2138997376 >> v1:172.21.15.64:6790/0 conn(0x7f2e900a98f0 legacy=0x7f2e900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:14.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.855+0000 7f2e8effd640 1 -- 172.21.15.64:0/2138997376 >> v1:172.21.15.64:6789/0 conn(0x7f2e9013f710 legacy=0x7f2e901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:14.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.855+0000 7f2e8effd640 10 monclient: _finish_hunting 0 2024-03-25T18:12:14.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.855+0000 7f2e8effd640 1 monclient: found mon.noname-c 2024-03-25T18:12:14.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.855+0000 7f2e8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:14.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.855+0000 7f2e8effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:12:14.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.855+0000 7f2e8effd640 1 -- 172.21.15.64:0/2138997376 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e900b44c0 con 0x7f2e901430c0 2024-03-25T18:12:14.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.855+0000 7f2e8effd640 10 monclient: _finish_auth 0 2024-03-25T18:12:14.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.855+0000 7f2e8effd640 10 monclient: _check_auth_tickets 2024-03-25T18:12:14.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.855+0000 7f2e8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:14.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.855+0000 7f2e8effd640 1 -- 172.21.15.64:0/2138997376 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2e88001a10 con 0x7f2e901430c0 2024-03-25T18:12:14.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.855+0000 7f2e8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:14.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.855+0000 7f2e8effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:14.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.855+0000 7f2e8effd640 10 monclient: dump: 2024-03-25T18:12:14.857 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:14.857 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:14.857 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:14.857 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:14.857 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:14.857 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:14.857 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:14.857 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:14.857 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:14.857 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:14.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.855+0000 7f2e9656d640 5 monclient: authenticate success, global_id 4333 2024-03-25T18:12:14.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.855+0000 7f2e9656d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:14.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.861+0000 7f2e8effd640 1 -- 172.21.15.64:0/2138997376 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e88001b70 con 0x7f2e901430c0 2024-03-25T18:12:14.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.861+0000 7f2e8effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:14.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.861+0000 7f2e8effd640 1 -- 172.21.15.64:0/2138997376 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2e88001f80 con 0x7f2e901430c0 2024-03-25T18:12:14.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.861+0000 7f2e8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:14.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.861+0000 7f2e8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:14.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.861+0000 7f2e8effd640 10 monclient: dump: 2024-03-25T18:12:14.862 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:14.863 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:14.863 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:14.863 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:14.863 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:14.863 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:14.863 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:14.863 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:14.863 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:14.863 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:14.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.861+0000 7f2e9656d640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:14.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.861+0000 7f2e9656d640 4 set_mon_vals no callback set 2024-03-25T18:12:14.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.862+0000 7f2e9656d640 10 monclient: shutdown 2024-03-25T18:12:14.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.862+0000 7f2e9656d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:14.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.862+0000 7f2e9656d640 1 -- 172.21.15.64:0/2138997376 >> v1:172.21.15.86:6789/0 conn(0x7f2e901430c0 legacy=0x7f2e9014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:14.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.862+0000 7f2e9656d640 1 -- 172.21.15.64:0/2138997376 shutdown_connections 2024-03-25T18:12:14.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.862+0000 7f2e9656d640 1 -- 172.21.15.64:0/2138997376 >> 172.21.15.64:0/2138997376 conn(0x7f2e900b7ea0 msgr2=0x7f2e900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:14.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.862+0000 7f2e9656d640 1 -- 172.21.15.64:0/2138997376 shutdown_connections 2024-03-25T18:12:14.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.862+0000 7f2e9656d640 1 -- 172.21.15.64:0/2138997376 wait complete. 2024-03-25T18:12:14.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.863+0000 7f2e9656d640 10 monclient: build_initial_monmap 2024-03-25T18:12:14.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.863+0000 7f2e9656d640 10 monclient: monmap: 2024-03-25T18:12:14.865 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:12:14.865 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:14.865 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:12:14.865 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:12:14.865 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:14.865 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:12:14.865 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:12:14.865 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:12:14.865 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:12:14.865 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:14.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.863+0000 7f2e9656d640 1 librados: starting msgr at 2024-03-25T18:12:14.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.863+0000 7f2e9656d640 1 librados: starting objecter 2024-03-25T18:12:14.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.864+0000 7f2e9656d640 1 Processor -- start 2024-03-25T18:12:14.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.864+0000 7f2e9656d640 1 -- start start 2024-03-25T18:12:14.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.864+0000 7f2e9656d640 1 librados: setting wanted keys 2024-03-25T18:12:14.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.864+0000 7f2e9656d640 1 librados: calling monclient init 2024-03-25T18:12:14.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.864+0000 7f2e9656d640 10 monclient: init 2024-03-25T18:12:14.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.864+0000 7f2e9656d640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:14.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.864+0000 7f2e9656d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:14.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.864+0000 7f2e9656d640 10 monclient(hunting): picked mon.noname-b con 0x7f2e900a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:12:14.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.864+0000 7f2e9656d640 10 monclient(hunting): picked mon.noname-c con 0x7f2e9013f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:12:14.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.864+0000 7f2e9656d640 10 monclient(hunting): picked mon.noname-a con 0x7f2e901430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:12:14.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.865+0000 7f2e9656d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e90136ec0 con 0x7f2e901430c0 2024-03-25T18:12:14.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.865+0000 7f2e9656d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e90136c20 con 0x7f2e900a98f0 2024-03-25T18:12:14.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.865+0000 7f2e9656d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e90137db0 con 0x7f2e9013f710 2024-03-25T18:12:14.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.865+0000 7f2e9656d640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:14.867 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.865+0000 7f2e9656d640 10 monclient(hunting): authenticate will time out at 7121.103536s 2024-03-25T18:12:14.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.865+0000 7f2e94b0a640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f2e901430c0 0x7f2e901e2c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:47128/0 (socket says 172.21.15.64:47128) 2024-03-25T18:12:14.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.865+0000 7f2e94b0a640 1 -- 172.21.15.64:0/662285179 learned_addr learned my addr 172.21.15.64:0/662285179 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:12:14.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.865+0000 7f2e8cff9640 10 client.?.objecter ms_handle_connect 0x7f2e901430c0 2024-03-25T18:12:14.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.865+0000 7f2e8cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:14.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.870+0000 7f2e8cff9640 10 client.?.objecter ms_handle_connect 0x7f2e900a98f0 2024-03-25T18:12:14.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.870+0000 7f2e8cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:14.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.929+0000 7f2e8cff9640 10 client.?.objecter ms_handle_connect 0x7f2e9013f710 2024-03-25T18:12:14.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.929+0000 7f2e8cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:14.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.939+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649205358 0 0) 0x7f2e90136ec0 con 0x7f2e901430c0 2024-03-25T18:12:14.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.939+0000 7f2e8cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:14.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.939+0000 7f2e8cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:14.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.939+0000 7f2e8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:14.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.939+0000 7f2e8cff9640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:12:14.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.939+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e7c004eb0 con 0x7f2e901430c0 2024-03-25T18:12:14.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.944+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104851946 0 0) 0x7f2e90136c20 con 0x7f2e900a98f0 2024-03-25T18:12:14.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.944+0000 7f2e8cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:14.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.944+0000 7f2e8cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:14.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.944+0000 7f2e8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:14.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.944+0000 7f2e8cff9640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:12:14.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.944+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e90136ec0 con 0x7f2e900a98f0 2024-03-25T18:12:14.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.944+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183072205 0 0) 0x7f2e90137db0 con 0x7f2e9013f710 2024-03-25T18:12:14.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.944+0000 7f2e8cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:14.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.944+0000 7f2e8cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:14.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.944+0000 7f2e8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:14.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.944+0000 7f2e8cff9640 10 monclient(hunting): my global_id is 4346 2024-03-25T18:12:14.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.944+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e90136c20 con 0x7f2e9013f710 2024-03-25T18:12:14.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.956+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3290049383 0 0) 0x7f2e90136c20 con 0x7f2e9013f710 2024-03-25T18:12:14.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.956+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e90137db0 con 0x7f2e9013f710 2024-03-25T18:12:14.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.956+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2e84001860 con 0x7f2e9013f710 2024-03-25T18:12:14.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.956+0000 7f2e8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:14.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.956+0000 7f2e8cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:14.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.956+0000 7f2e8cff9640 10 monclient(hunting): dump: 2024-03-25T18:12:14.957 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:14.957 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:14.957 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:14.958 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:14.958 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:14.958 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:14.958 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:14.958 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:14.958 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:14.958 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:14.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.956+0000 7f2e8cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:14.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.963+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3964961109 0 0) 0x7f2e90136ec0 con 0x7f2e900a98f0 2024-03-25T18:12:14.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.963+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e90136c20 con 0x7f2e900a98f0 2024-03-25T18:12:14.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.963+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2e80001930 con 0x7f2e900a98f0 2024-03-25T18:12:14.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.963+0000 7f2e8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:14.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.963+0000 7f2e8cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:14.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.963+0000 7f2e8cff9640 10 monclient(hunting): dump: 2024-03-25T18:12:14.964 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:14.964 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:14.964 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:14.964 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:14.964 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:14.964 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:14.964 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:14.965 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:14.965 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:14.965 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:14.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.967+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 125127485 0 0) 0x7f2e7c004eb0 con 0x7f2e901430c0 2024-03-25T18:12:14.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.967+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e90136ec0 con 0x7f2e901430c0 2024-03-25T18:12:14.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.967+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2e88001f40 con 0x7f2e901430c0 2024-03-25T18:12:14.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.967+0000 7f2e8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:14.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.967+0000 7f2e8cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:14.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.967+0000 7f2e8cff9640 10 monclient(hunting): dump: 2024-03-25T18:12:14.968 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:14.968 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:14.968 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:14.968 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:14.968 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:14.969 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:14.969 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:14.969 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:14.969 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:14.969 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:14.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2663482349 0 0) 0x7f2e90136c20 con 0x7f2e900a98f0 2024-03-25T18:12:14.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 >> v1:172.21.15.64:6790/0 conn(0x7f2e9013f710 legacy=0x7f2e900b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:14.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 >> v1:172.21.15.64:6789/0 conn(0x7f2e901430c0 legacy=0x7f2e901e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:14.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e8cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:12:14.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e8cff9640 1 monclient: found mon.b 2024-03-25T18:12:14.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e8cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:14.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e8cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:12:14.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e90138b50 con 0x7f2e900a98f0 2024-03-25T18:12:14.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e8cff9640 10 monclient: _finish_auth 0 2024-03-25T18:12:14.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e8cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:12:14.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:14.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e9656d640 5 monclient: authenticate success, global_id 4336 2024-03-25T18:12:14.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e9656d640 10 monclient: _renew_subs 2024-03-25T18:12:14.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e9656d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:12:14.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e9656d640 1 -- 172.21.15.64:0/662285179 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e90139830 con 0x7f2e900a98f0 2024-03-25T18:12:14.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e9656d640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:14.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e9656d640 10 monclient: _renew_subs 2024-03-25T18:12:14.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e9656d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:12:14.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e9656d640 1 -- 172.21.15.64:0/662285179 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e901369b0 con 0x7f2e900a98f0 2024-03-25T18:12:14.971 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.969+0000 7f2e9656d640 1 librados: init done 2024-03-25T18:12:14.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.970+0000 7f2e9656d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:14.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.970+0000 7f2e9656d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:14.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.970+0000 7f2e9656d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:12:14.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:14.970+0000 7f2e9656d640 1 -- 172.21.15.64:0/662285179 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e90139a20 con 0x7f2e900a98f0 2024-03-25T18:12:15.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.236+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e800035f0 con 0x7f2e900a98f0 2024-03-25T18:12:15.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.236+0000 7f2e8cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:15.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.236+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2e80003760 con 0x7f2e900a98f0 2024-03-25T18:12:15.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.236+0000 7f2e8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:15.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.236+0000 7f2e8cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:15.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.236+0000 7f2e8cff9640 10 monclient: dump: 2024-03-25T18:12:15.237 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:15.238 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:15.238 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:15.238 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:15.238 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:15.238 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:15.238 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:15.238 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:15.238 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:15.238 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:15.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.236+0000 7f2e8e7fc640 4 set_mon_vals no callback set 2024-03-25T18:12:15.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.237+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f2e80019730 con 0x7f2e900a98f0 2024-03-25T18:12:15.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.238+0000 7f2e8f7fe640 10 monclient: get_auth_request con 0x7f2e7c06e110 auth_method 0 2024-03-25T18:12:15.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.238+0000 7f2e8cff9640 10 client.4336.objecter ms_handle_connect 0x7f2e7c06e110 2024-03-25T18:12:15.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.243+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f2e80053340 con 0x7f2e900a98f0 2024-03-25T18:12:15.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.243+0000 7f2e8cff9640 10 client.4336.objecter ms_dispatch 0x7f2e90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:15.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.243+0000 7f2e8cff9640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:15.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.243+0000 7f2e8cff9640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:15.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.243+0000 7f2e8cff9640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T18:12:15.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.243+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e8004ee50 con 0x7f2e900a98f0 2024-03-25T18:12:15.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.243+0000 7f2e8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:15.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.243+0000 7f2e8cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:15.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.506+0000 7f2e9656d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:15.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.506+0000 7f2e9656d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:15.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.506+0000 7f2e9656d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:12:15.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.506+0000 7f2e9656d640 1 -- 172.21.15.64:0/662285179 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e900627b0 con 0x7f2e900a98f0 2024-03-25T18:12:15.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.533+0000 7f2e8cff9640 1 -- 172.21.15.64:0/662285179 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f2e80050d80 con 0x7f2e900a98f0 2024-03-25T18:12:15.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.533+0000 7f2e8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:15.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.533+0000 7f2e8cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:15.534 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:12:15.534 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:15.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.535+0000 7f2e9656d640 10 librados: watch_flush enter 2024-03-25T18:12:15.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.535+0000 7f2e9656d640 10 librados: watch_flush exit 2024-03-25T18:12:15.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.535+0000 7f2e9656d640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T18:12:15.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.535+0000 7f2e9656d640 10 client.4336.objecter successfully canceled tick 2024-03-25T18:12:15.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.535+0000 7f2e9656d640 1 -- 172.21.15.64:0/662285179 >> v1:172.21.15.64:6800/2915543638 conn(0x7f2e7c06e110 legacy=0x7f2e7c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:15.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.535+0000 7f2e9656d640 10 monclient: shutdown 2024-03-25T18:12:15.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.535+0000 7f2e9656d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:15.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.535+0000 7f2e9656d640 1 -- 172.21.15.64:0/662285179 >> v1:172.21.15.86:6789/0 conn(0x7f2e900a98f0 legacy=0x7f2e901376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:15.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.535+0000 7f2e9656d640 1 -- 172.21.15.64:0/662285179 shutdown_connections 2024-03-25T18:12:15.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.535+0000 7f2e9656d640 1 -- 172.21.15.64:0/662285179 >> 172.21.15.64:0/662285179 conn(0x7f2e900b7ea0 msgr2=0x7f2e9013ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:15.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.536+0000 7f2e9656d640 1 -- 172.21.15.64:0/662285179 shutdown_connections 2024-03-25T18:12:15.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.536+0000 7f2e9656d640 1 -- 172.21.15.64:0/662285179 wait complete. 2024-03-25T18:12:15.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:15.536+0000 7f2e9656d640 1 librados: shutdown 2024-03-25T18:12:15.549 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:22.355 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:22.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.466+0000 7fe21160d640 10 monclient: get_monmap_and_config 2024-03-25T18:12:22.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.466+0000 7fe21160d640 10 monclient: build_initial_monmap 2024-03-25T18:12:22.467 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.466+0000 7fe21160d640 10 monclient: monmap: 2024-03-25T18:12:22.467 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:12:22.467 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:22.467 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:12:22.467430+0000 2024-03-25T18:12:22.467 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:12:22.467430+0000 2024-03-25T18:12:22.467 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:22.467 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:22.467 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:12:22.468 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:12:22.468 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:12:22.468 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:22.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.467+0000 7fe21160d640 1 Processor -- start 2024-03-25T18:12:22.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.467+0000 7fe21160d640 1 -- start start 2024-03-25T18:12:22.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.467+0000 7fe21160d640 10 monclient: init 2024-03-25T18:12:22.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.468+0000 7fe21160d640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:22.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.468+0000 7fe21160d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:22.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.468+0000 7fe21160d640 10 monclient(hunting): picked mon.noname-c con 0x7fe20c0a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:12:22.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.468+0000 7fe21160d640 10 monclient(hunting): picked mon.noname-a con 0x7fe20c13f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:12:22.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.468+0000 7fe21160d640 10 monclient(hunting): picked mon.noname-b con 0x7fe20c1430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:12:22.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.468+0000 7fe21160d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c142520 con 0x7fe20c13f710 2024-03-25T18:12:22.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.468+0000 7fe21160d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c0b28a0 con 0x7fe20c0a98f0 2024-03-25T18:12:22.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.468+0000 7fe21160d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c0b2a30 con 0x7fe20c1430c0 2024-03-25T18:12:22.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.468+0000 7fe21160d640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:22.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.468+0000 7fe21160d640 10 monclient(hunting): authenticate will time out at 7128.706844s 2024-03-25T18:12:22.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.477+0000 7fe20b7fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe20c1430c0 0x7fe20c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:38998/0 (socket says 172.21.15.64:38998) 2024-03-25T18:12:22.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.477+0000 7fe20b7fe640 1 -- 172.21.15.64:0/1973986809 learned_addr learned my addr 172.21.15.64:0/1973986809 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:12:22.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.488+0000 7fe209ffb640 1 -- 172.21.15.64:0/1973986809 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252878583 0 0) 0x7fe20c0b28a0 con 0x7fe20c0a98f0 2024-03-25T18:12:22.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.488+0000 7fe209ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:22.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.488+0000 7fe209ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:22.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.488+0000 7fe209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:22.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.488+0000 7fe209ffb640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:12:22.489 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.488+0000 7fe209ffb640 1 -- 172.21.15.64:0/1973986809 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f4003520 con 0x7fe20c0a98f0 2024-03-25T18:12:22.493 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.492+0000 7fe209ffb640 1 -- 172.21.15.64:0/1973986809 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359886392 0 0) 0x7fe20c0b2a30 con 0x7fe20c1430c0 2024-03-25T18:12:22.493 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.492+0000 7fe209ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:22.493 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.492+0000 7fe209ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:22.493 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.492+0000 7fe209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:22.493 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.492+0000 7fe209ffb640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:12:22.493 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.492+0000 7fe209ffb640 1 -- 172.21.15.64:0/1973986809 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe20c0b28a0 con 0x7fe20c1430c0 2024-03-25T18:12:22.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.508+0000 7fe209ffb640 1 -- 172.21.15.64:0/1973986809 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424561766 0 0) 0x7fe20c142520 con 0x7fe20c13f710 2024-03-25T18:12:22.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.508+0000 7fe209ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:22.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.508+0000 7fe209ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:22.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.508+0000 7fe209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:22.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.508+0000 7fe209ffb640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:12:22.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.508+0000 7fe209ffb640 1 -- 172.21.15.64:0/1973986809 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe20c0b2a30 con 0x7fe20c13f710 2024-03-25T18:12:22.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.512+0000 7fe209ffb640 1 -- 172.21.15.64:0/1973986809 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 312952632 0 0) 0x7fe1f4003520 con 0x7fe20c0a98f0 2024-03-25T18:12:22.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.512+0000 7fe209ffb640 1 -- 172.21.15.64:0/1973986809 >> v1:172.21.15.64:6790/0 conn(0x7fe20c1430c0 legacy=0x7fe20c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:22.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.512+0000 7fe209ffb640 1 -- 172.21.15.64:0/1973986809 >> v1:172.21.15.64:6789/0 conn(0x7fe20c13f710 legacy=0x7fe20c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:22.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.512+0000 7fe209ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:12:22.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.512+0000 7fe209ffb640 1 monclient: found mon.noname-c 2024-03-25T18:12:22.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.512+0000 7fe209ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:22.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.512+0000 7fe209ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:12:22.513 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.512+0000 7fe209ffb640 1 -- 172.21.15.64:0/1973986809 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c0b44c0 con 0x7fe20c0a98f0 2024-03-25T18:12:22.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.512+0000 7fe209ffb640 10 monclient: _finish_auth 0 2024-03-25T18:12:22.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.512+0000 7fe209ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:12:22.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.512+0000 7fe209ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:22.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.512+0000 7fe209ffb640 1 -- 172.21.15.64:0/1973986809 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe1fc001690 con 0x7fe20c0a98f0 2024-03-25T18:12:22.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.512+0000 7fe21160d640 5 monclient: authenticate success, global_id 4339 2024-03-25T18:12:22.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.512+0000 7fe21160d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:22.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.512+0000 7fe209ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:22.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.512+0000 7fe209ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:22.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.512+0000 7fe209ffb640 10 monclient: dump: 2024-03-25T18:12:22.514 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:22.514 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:22.514 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:22.515 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:22.515 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:22.515 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:22.515 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:22.515 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:22.515 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:22.515 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:22.515 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.512+0000 7fe21160d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:22.530 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.529+0000 7fe209ffb640 1 -- 172.21.15.64:0/1973986809 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1fc0017f0 con 0x7fe20c0a98f0 2024-03-25T18:12:22.530 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.529+0000 7fe209ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:22.530 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.529+0000 7fe209ffb640 1 -- 172.21.15.64:0/1973986809 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe1fc001c00 con 0x7fe20c0a98f0 2024-03-25T18:12:22.530 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.529+0000 7fe209ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:22.530 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.529+0000 7fe209ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:22.530 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.529+0000 7fe209ffb640 10 monclient: dump: 2024-03-25T18:12:22.530 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:22.530 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:22.530 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:22.530 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:22.530 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:22.530 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:22.530 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:22.530 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:22.531 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:22.531 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:22.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.529+0000 7fe21160d640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:22.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.529+0000 7fe21160d640 4 set_mon_vals no callback set 2024-03-25T18:12:22.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.530+0000 7fe21160d640 10 monclient: shutdown 2024-03-25T18:12:22.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.530+0000 7fe21160d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:22.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.530+0000 7fe21160d640 1 -- 172.21.15.64:0/1973986809 >> v1:172.21.15.86:6789/0 conn(0x7fe20c0a98f0 legacy=0x7fe20c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:22.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.530+0000 7fe21160d640 1 -- 172.21.15.64:0/1973986809 shutdown_connections 2024-03-25T18:12:22.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.530+0000 7fe21160d640 1 -- 172.21.15.64:0/1973986809 >> 172.21.15.64:0/1973986809 conn(0x7fe20c0b7ea0 msgr2=0x7fe20c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:22.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.530+0000 7fe21160d640 1 -- 172.21.15.64:0/1973986809 shutdown_connections 2024-03-25T18:12:22.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.530+0000 7fe21160d640 1 -- 172.21.15.64:0/1973986809 wait complete. 2024-03-25T18:12:22.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.530+0000 7fe21160d640 10 monclient: build_initial_monmap 2024-03-25T18:12:22.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.530+0000 7fe21160d640 10 monclient: monmap: 2024-03-25T18:12:22.533 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:12:22.533 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:22.533 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:12:22.533 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:12:22.533 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:22.533 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:12:22.534 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:12:22.534 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:12:22.534 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:12:22.534 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:22.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.531+0000 7fe21160d640 1 librados: starting msgr at 2024-03-25T18:12:22.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.531+0000 7fe21160d640 1 librados: starting objecter 2024-03-25T18:12:22.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.531+0000 7fe21160d640 1 Processor -- start 2024-03-25T18:12:22.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.531+0000 7fe21160d640 1 -- start start 2024-03-25T18:12:22.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.531+0000 7fe21160d640 1 librados: setting wanted keys 2024-03-25T18:12:22.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.531+0000 7fe21160d640 1 librados: calling monclient init 2024-03-25T18:12:22.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.531+0000 7fe21160d640 10 monclient: init 2024-03-25T18:12:22.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.532+0000 7fe21160d640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:22.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.532+0000 7fe21160d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:22.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.532+0000 7fe21160d640 10 monclient(hunting): picked mon.noname-c con 0x7fe20c0a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:12:22.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.532+0000 7fe21160d640 10 monclient(hunting): picked mon.noname-a con 0x7fe20c13f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:12:22.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.532+0000 7fe21160d640 10 monclient(hunting): picked mon.noname-b con 0x7fe20c1430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:12:22.535 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.532+0000 7fe21160d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c137ab0 con 0x7fe20c13f710 2024-03-25T18:12:22.535 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.532+0000 7fe21160d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c138bd0 con 0x7fe20c1430c0 2024-03-25T18:12:22.535 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.532+0000 7fe21160d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe20c138e40 con 0x7fe20c0a98f0 2024-03-25T18:12:22.535 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.532+0000 7fe21160d640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:22.535 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.532+0000 7fe21160d640 10 monclient(hunting): authenticate will time out at 7128.770774s 2024-03-25T18:12:22.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.536+0000 7fe20a7fc640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fe20c13f710 0x7fe20c1df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:60914/0 (socket says 172.21.15.64:60914) 2024-03-25T18:12:22.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.537+0000 7fe20a7fc640 1 -- 172.21.15.64:0/1050357198 learned_addr learned my addr 172.21.15.64:0/1050357198 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:12:22.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.541+0000 7fe1f37fe640 10 client.?.objecter ms_handle_connect 0x7fe20c13f710 2024-03-25T18:12:22.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.541+0000 7fe1f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:22.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.548+0000 7fe1f37fe640 10 client.?.objecter ms_handle_connect 0x7fe20c1430c0 2024-03-25T18:12:22.549 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.548+0000 7fe1f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:22.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.554+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739849947 0 0) 0x7fe20c137ab0 con 0x7fe20c13f710 2024-03-25T18:12:22.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.554+0000 7fe1f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:22.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.554+0000 7fe1f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:22.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.554+0000 7fe1f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:22.555 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.554+0000 7fe1f37fe640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:12:22.555 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.554+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f8004eb0 con 0x7fe20c13f710 2024-03-25T18:12:22.555 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.555+0000 7fe1f37fe640 10 client.?.objecter ms_handle_connect 0x7fe20c0a98f0 2024-03-25T18:12:22.555 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.555+0000 7fe1f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:22.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.563+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1202252737 0 0) 0x7fe1f8004eb0 con 0x7fe20c13f710 2024-03-25T18:12:22.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.563+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe20c137ab0 con 0x7fe20c13f710 2024-03-25T18:12:22.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.563+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe200001a10 con 0x7fe20c13f710 2024-03-25T18:12:22.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.563+0000 7fe1f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:22.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.563+0000 7fe1f37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:22.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.563+0000 7fe1f37fe640 10 monclient(hunting): dump: 2024-03-25T18:12:22.564 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:22.564 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:22.565 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:22.565 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:22.565 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:22.565 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:22.565 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:22.565 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:22.565 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:22.565 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:22.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.563+0000 7fe1f37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:22.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.565+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695393101 0 0) 0x7fe20c138bd0 con 0x7fe20c1430c0 2024-03-25T18:12:22.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.565+0000 7fe1f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:22.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.565+0000 7fe1f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:22.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.565+0000 7fe1f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:22.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.565+0000 7fe1f37fe640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:12:22.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.565+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f8004eb0 con 0x7fe20c1430c0 2024-03-25T18:12:22.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.569+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1687775770 0 0) 0x7fe20c138e40 con 0x7fe20c0a98f0 2024-03-25T18:12:22.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.569+0000 7fe1f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:22.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.569+0000 7fe1f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:22.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.569+0000 7fe1f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:22.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.569+0000 7fe1f37fe640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:12:22.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.569+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe20c138bd0 con 0x7fe20c0a98f0 2024-03-25T18:12:22.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4289856478 0 0) 0x7fe20c137ab0 con 0x7fe20c13f710 2024-03-25T18:12:22.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 >> v1:172.21.15.64:6790/0 conn(0x7fe20c0a98f0 legacy=0x7fe20c137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:22.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 >> v1:172.21.15.86:6789/0 conn(0x7fe20c1430c0 legacy=0x7fe20c1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:22.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe1f37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:12:22.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe1f37fe640 1 monclient: found mon.a 2024-03-25T18:12:22.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe1f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:22.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe1f37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:12:22.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20c139120 con 0x7fe20c13f710 2024-03-25T18:12:22.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe1f37fe640 10 monclient: _finish_auth 0 2024-03-25T18:12:22.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe1f37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:12:22.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe1f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:22.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe21160d640 5 monclient: authenticate success, global_id 4347 2024-03-25T18:12:22.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe21160d640 10 monclient: _renew_subs 2024-03-25T18:12:22.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe21160d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:12:22.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe21160d640 1 -- 172.21.15.64:0/1050357198 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe20c139400 con 0x7fe20c13f710 2024-03-25T18:12:22.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe21160d640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:22.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe21160d640 10 monclient: _renew_subs 2024-03-25T18:12:22.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe21160d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:12:22.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe21160d640 1 -- 172.21.15.64:0/1050357198 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe20c0b7880 con 0x7fe20c13f710 2024-03-25T18:12:22.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.573+0000 7fe21160d640 1 librados: init done 2024-03-25T18:12:22.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.575+0000 7fe21160d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:22.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.575+0000 7fe21160d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:22.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.575+0000 7fe21160d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:12:22.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.575+0000 7fe21160d640 1 -- 172.21.15.64:0/1050357198 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe20c137810 con 0x7fe20c13f710 2024-03-25T18:12:22.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.894+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe200001cb0 con 0x7fe20c13f710 2024-03-25T18:12:22.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.894+0000 7fe1f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:22.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.894+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe200003a60 con 0x7fe20c13f710 2024-03-25T18:12:22.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.894+0000 7fe1f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:22.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.894+0000 7fe1f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:22.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.894+0000 7fe1f37fe640 10 monclient: dump: 2024-03-25T18:12:22.896 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:22.896 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:22.896 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:22.896 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:22.896 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:22.896 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:22.896 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:22.896 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:22.896 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:22.896 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:22.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.894+0000 7fe2097fa640 4 set_mon_vals no callback set 2024-03-25T18:12:22.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.895+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fe200003c80 con 0x7fe20c13f710 2024-03-25T18:12:22.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.896+0000 7fe20affd640 10 monclient: get_auth_request con 0x7fe1f806de80 auth_method 0 2024-03-25T18:12:22.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.896+0000 7fe1f37fe640 10 client.4347.objecter ms_handle_connect 0x7fe1f806de80 2024-03-25T18:12:22.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.901+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fe200054890 con 0x7fe20c13f710 2024-03-25T18:12:22.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.901+0000 7fe1f37fe640 10 client.4347.objecter ms_dispatch 0x7fe20c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:22.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.901+0000 7fe1f37fe640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:22.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.901+0000 7fe1f37fe640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:22.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.901+0000 7fe1f37fe640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-25T18:12:22.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.901+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe2000503a0 con 0x7fe20c13f710 2024-03-25T18:12:22.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.901+0000 7fe1f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:22.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:22.901+0000 7fe1f37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:23.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.138+0000 7fe21160d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:23.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.138+0000 7fe21160d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:23.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.138+0000 7fe21160d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:12:23.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.138+0000 7fe21160d640 1 -- 172.21.15.64:0/1050357198 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe20c137c60 con 0x7fe20c13f710 2024-03-25T18:12:23.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.173+0000 7fe1f37fe640 1 -- 172.21.15.64:0/1050357198 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fe20c137c60 con 0x7fe20c13f710 2024-03-25T18:12:23.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.173+0000 7fe1f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:23.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.173+0000 7fe1f37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:23.174 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:12:23.175 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:23.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.176+0000 7fe21160d640 10 librados: watch_flush enter 2024-03-25T18:12:23.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.176+0000 7fe21160d640 10 librados: watch_flush exit 2024-03-25T18:12:23.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.176+0000 7fe21160d640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-25T18:12:23.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.176+0000 7fe21160d640 10 client.4347.objecter successfully canceled tick 2024-03-25T18:12:23.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.176+0000 7fe21160d640 1 -- 172.21.15.64:0/1050357198 >> v1:172.21.15.64:6800/2915543638 conn(0x7fe1f806de80 legacy=0x7fe1f8070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:23.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.176+0000 7fe21160d640 10 monclient: shutdown 2024-03-25T18:12:23.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.176+0000 7fe21160d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:23.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.176+0000 7fe21160d640 1 -- 172.21.15.64:0/1050357198 >> v1:172.21.15.64:6789/0 conn(0x7fe20c13f710 legacy=0x7fe20c1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:23.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.176+0000 7fe21160d640 1 -- 172.21.15.64:0/1050357198 shutdown_connections 2024-03-25T18:12:23.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.176+0000 7fe21160d640 1 -- 172.21.15.64:0/1050357198 >> 172.21.15.64:0/1050357198 conn(0x7fe20c0b7ea0 msgr2=0x7fe20c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:23.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.176+0000 7fe21160d640 1 -- 172.21.15.64:0/1050357198 shutdown_connections 2024-03-25T18:12:23.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.176+0000 7fe21160d640 1 -- 172.21.15.64:0/1050357198 wait complete. 2024-03-25T18:12:23.178 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:23.176+0000 7fe21160d640 1 librados: shutdown 2024-03-25T18:12:23.188 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:29.996 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:30.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.110+0000 7f467ec25640 10 monclient: get_monmap_and_config 2024-03-25T18:12:30.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.110+0000 7f467ec25640 10 monclient: build_initial_monmap 2024-03-25T18:12:30.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.110+0000 7f467ec25640 10 monclient: monmap: 2024-03-25T18:12:30.111 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:12:30.111 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:30.111 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:12:30.111449+0000 2024-03-25T18:12:30.111 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:12:30.111449+0000 2024-03-25T18:12:30.111 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:30.112 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:30.112 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:12:30.112 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:12:30.112 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:12:30.112 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:30.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.111+0000 7f467ec25640 1 Processor -- start 2024-03-25T18:12:30.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.111+0000 7f467ec25640 1 -- start start 2024-03-25T18:12:30.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.111+0000 7f467ec25640 10 monclient: init 2024-03-25T18:12:30.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.112+0000 7f467ec25640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:30.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.112+0000 7f467ec25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:30.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.112+0000 7f467ec25640 10 monclient(hunting): picked mon.noname-b con 0x7f467806ab80 addr v1:172.21.15.64:6790/0 2024-03-25T18:12:30.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.112+0000 7f467ec25640 10 monclient(hunting): picked mon.noname-a con 0x7f467806af60 addr v1:172.21.15.64:6789/0 2024-03-25T18:12:30.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.112+0000 7f467ec25640 10 monclient(hunting): picked mon.noname-c con 0x7f467813f990 addr v1:172.21.15.86:6789/0 2024-03-25T18:12:30.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.112+0000 7f467ec25640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46780afa60 con 0x7f467806af60 2024-03-25T18:12:30.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.112+0000 7f467ec25640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46780afce0 con 0x7f467813f990 2024-03-25T18:12:30.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.112+0000 7f467ec25640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46780b1150 con 0x7f467806ab80 2024-03-25T18:12:30.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.112+0000 7f467ec25640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:30.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.112+0000 7f467ec25640 10 monclient(hunting): authenticate will time out at 7136.351201s 2024-03-25T18:12:30.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.115+0000 7f467d1c2640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f467813f990 0x7f467814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:35862/0 (socket says 172.21.15.64:35862) 2024-03-25T18:12:30.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.115+0000 7f467d1c2640 1 -- 172.21.15.64:0/1055340363 learned_addr learned my addr 172.21.15.64:0/1055340363 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:12:30.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.125+0000 7f46777fe640 1 -- 172.21.15.64:0/1055340363 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993639952 0 0) 0x7f46780afce0 con 0x7f467813f990 2024-03-25T18:12:30.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.125+0000 7f46777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:30.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.125+0000 7f46777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:30.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.125+0000 7f46777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:30.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.125+0000 7f46777fe640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:12:30.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.125+0000 7f46777fe640 1 -- 172.21.15.64:0/1055340363 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4668003520 con 0x7f467813f990 2024-03-25T18:12:30.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.148+0000 7f46777fe640 1 -- 172.21.15.64:0/1055340363 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769497610 0 0) 0x7f46780afa60 con 0x7f467806af60 2024-03-25T18:12:30.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.148+0000 7f46777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:30.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.148+0000 7f46777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:30.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.148+0000 7f46777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:30.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.148+0000 7f46777fe640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:12:30.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.148+0000 7f46777fe640 1 -- 172.21.15.64:0/1055340363 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46780afce0 con 0x7f467806af60 2024-03-25T18:12:30.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.153+0000 7f46777fe640 1 -- 172.21.15.64:0/1055340363 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4225763659 0 0) 0x7f4668003520 con 0x7f467813f990 2024-03-25T18:12:30.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.153+0000 7f46777fe640 1 -- 172.21.15.64:0/1055340363 >> v1:172.21.15.64:6790/0 conn(0x7f467806ab80 legacy=0x7f46780724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:30.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.153+0000 7f46777fe640 1 -- 172.21.15.64:0/1055340363 >> v1:172.21.15.64:6789/0 conn(0x7f467806af60 legacy=0x7f46781458c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:30.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.153+0000 7f46777fe640 10 monclient: _finish_hunting 0 2024-03-25T18:12:30.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.153+0000 7f46777fe640 1 monclient: found mon.noname-c 2024-03-25T18:12:30.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.153+0000 7f46777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:30.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.153+0000 7f46777fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:12:30.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.153+0000 7f46777fe640 1 -- 172.21.15.64:0/1055340363 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46780aff20 con 0x7f467813f990 2024-03-25T18:12:30.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.153+0000 7f46777fe640 10 monclient: _finish_auth 0 2024-03-25T18:12:30.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.153+0000 7f46777fe640 10 monclient: _check_auth_tickets 2024-03-25T18:12:30.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.153+0000 7f46777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:30.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.153+0000 7f46777fe640 1 -- 172.21.15.64:0/1055340363 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4670001a10 con 0x7f467813f990 2024-03-25T18:12:30.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.153+0000 7f467ec25640 5 monclient: authenticate success, global_id 4345 2024-03-25T18:12:30.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.153+0000 7f467ec25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:30.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.153+0000 7f46777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:30.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.153+0000 7f46777fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:30.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.153+0000 7f46777fe640 10 monclient: dump: 2024-03-25T18:12:30.155 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:30.155 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:30.155 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:30.155 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:30.155 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:30.155 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:30.155 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:30.156 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:30.156 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:30.156 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:30.156 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.153+0000 7f467ec25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:30.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.166+0000 7f46777fe640 1 -- 172.21.15.64:0/1055340363 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4670001b70 con 0x7f467813f990 2024-03-25T18:12:30.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.166+0000 7f46777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:30.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.166+0000 7f46777fe640 1 -- 172.21.15.64:0/1055340363 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4670001f80 con 0x7f467813f990 2024-03-25T18:12:30.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.166+0000 7f46777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:30.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.166+0000 7f46777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:30.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.166+0000 7f46777fe640 10 monclient: dump: 2024-03-25T18:12:30.167 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:30.167 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:30.167 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:30.167 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:30.167 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:30.167 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:30.167 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:30.167 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:30.168 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:30.168 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:30.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.166+0000 7f467ec25640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:30.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.166+0000 7f467ec25640 4 set_mon_vals no callback set 2024-03-25T18:12:30.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.167+0000 7f467ec25640 10 monclient: shutdown 2024-03-25T18:12:30.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.167+0000 7f467ec25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:30.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.167+0000 7f467ec25640 1 -- 172.21.15.64:0/1055340363 >> v1:172.21.15.86:6789/0 conn(0x7f467813f990 legacy=0x7f467814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:30.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.167+0000 7f467ec25640 1 -- 172.21.15.64:0/1055340363 shutdown_connections 2024-03-25T18:12:30.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.167+0000 7f467ec25640 1 -- 172.21.15.64:0/1055340363 >> 172.21.15.64:0/1055340363 conn(0x7f467806c830 msgr2=0x7f46780b4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:30.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.167+0000 7f467ec25640 1 -- 172.21.15.64:0/1055340363 shutdown_connections 2024-03-25T18:12:30.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.167+0000 7f467ec25640 1 -- 172.21.15.64:0/1055340363 wait complete. 2024-03-25T18:12:30.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.168+0000 7f467ec25640 10 monclient: build_initial_monmap 2024-03-25T18:12:30.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.168+0000 7f467ec25640 10 monclient: monmap: 2024-03-25T18:12:30.170 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:12:30.170 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:30.170 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:12:30.170 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:12:30.170 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:30.170 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:12:30.170 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:12:30.170 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:12:30.170 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:12:30.170 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:30.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.168+0000 7f467ec25640 1 librados: starting msgr at 2024-03-25T18:12:30.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.168+0000 7f467ec25640 1 librados: starting objecter 2024-03-25T18:12:30.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.168+0000 7f467ec25640 1 Processor -- start 2024-03-25T18:12:30.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.168+0000 7f467ec25640 1 -- start start 2024-03-25T18:12:30.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.168+0000 7f467ec25640 1 librados: setting wanted keys 2024-03-25T18:12:30.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.168+0000 7f467ec25640 1 librados: calling monclient init 2024-03-25T18:12:30.171 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.168+0000 7f467ec25640 10 monclient: init 2024-03-25T18:12:30.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.169+0000 7f467ec25640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:30.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.169+0000 7f467ec25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:30.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.169+0000 7f467ec25640 10 monclient(hunting): picked mon.noname-c con 0x7f467806ab80 addr v1:172.21.15.64:6790/0 2024-03-25T18:12:30.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.169+0000 7f467ec25640 10 monclient(hunting): picked mon.noname-b con 0x7f467813f990 addr v1:172.21.15.86:6789/0 2024-03-25T18:12:30.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.169+0000 7f467ec25640 10 monclient(hunting): picked mon.noname-a con 0x7f46781d41b0 addr v1:172.21.15.64:6789/0 2024-03-25T18:12:30.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.169+0000 7f467ec25640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467813e8c0 con 0x7f46781d41b0 2024-03-25T18:12:30.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.169+0000 7f467ec25640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467813eba0 con 0x7f467813f990 2024-03-25T18:12:30.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.169+0000 7f467ec25640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467813ee80 con 0x7f467806ab80 2024-03-25T18:12:30.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.169+0000 7f467ec25640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:30.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.169+0000 7f467ec25640 10 monclient(hunting): authenticate will time out at 7136.408159s 2024-03-25T18:12:30.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.185+0000 7f467c9c1640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f467806ab80 0x7f467813d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:51860/0 (socket says 172.21.15.64:51860) 2024-03-25T18:12:30.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.185+0000 7f467c9c1640 1 -- 172.21.15.64:0/3074259887 learned_addr learned my addr 172.21.15.64:0/3074259887 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:12:30.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.186+0000 7f46757fa640 10 client.?.objecter ms_handle_connect 0x7f467806ab80 2024-03-25T18:12:30.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.186+0000 7f46757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:30.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.194+0000 7f46757fa640 10 client.?.objecter ms_handle_connect 0x7f46781d41b0 2024-03-25T18:12:30.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.194+0000 7f46757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:30.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.195+0000 7f46757fa640 10 client.?.objecter ms_handle_connect 0x7f467813f990 2024-03-25T18:12:30.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.195+0000 7f46757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:30.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.203+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948205851 0 0) 0x7f467813ee80 con 0x7f467806ab80 2024-03-25T18:12:30.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.203+0000 7f46757fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:30.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.203+0000 7f46757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:30.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.203+0000 7f46757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:30.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.203+0000 7f46757fa640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:12:30.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.203+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f465c004eb0 con 0x7f467806ab80 2024-03-25T18:12:30.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.211+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754730796 0 0) 0x7f467813e8c0 con 0x7f46781d41b0 2024-03-25T18:12:30.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.211+0000 7f46757fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:30.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.211+0000 7f46757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:30.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.211+0000 7f46757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:30.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.211+0000 7f46757fa640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:12:30.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.211+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f467813ee80 con 0x7f46781d41b0 2024-03-25T18:12:30.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.217+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165016895 0 0) 0x7f467813eba0 con 0x7f467813f990 2024-03-25T18:12:30.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.217+0000 7f46757fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:30.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.217+0000 7f46757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:30.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.217+0000 7f46757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:30.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.217+0000 7f46757fa640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:12:30.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.217+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f467813e8c0 con 0x7f467813f990 2024-03-25T18:12:30.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.223+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2663936145 0 0) 0x7f465c004eb0 con 0x7f467806ab80 2024-03-25T18:12:30.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.223+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f467813eba0 con 0x7f467806ab80 2024-03-25T18:12:30.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.223+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f466c001a30 con 0x7f467806ab80 2024-03-25T18:12:30.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.223+0000 7f46757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:30.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.223+0000 7f46757fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:30.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.223+0000 7f46757fa640 10 monclient(hunting): dump: 2024-03-25T18:12:30.224 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:30.224 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:30.224 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:30.224 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:30.225 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:30.225 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:30.225 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:30.225 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:30.225 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:30.225 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:30.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.223+0000 7f46757fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:30.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.230+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3126923594 0 0) 0x7f467813e8c0 con 0x7f467813f990 2024-03-25T18:12:30.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.230+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f465c004eb0 con 0x7f467813f990 2024-03-25T18:12:30.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.230+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4664001930 con 0x7f467813f990 2024-03-25T18:12:30.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.230+0000 7f46757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:30.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.230+0000 7f46757fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:30.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.230+0000 7f46757fa640 10 monclient(hunting): dump: 2024-03-25T18:12:30.231 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:30.231 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:30.231 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:30.231 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:30.231 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:30.232 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:30.232 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:30.232 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:30.232 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:30.232 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:30.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.235+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2031119027 0 0) 0x7f467813ee80 con 0x7f46781d41b0 2024-03-25T18:12:30.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.235+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f467813e8c0 con 0x7f46781d41b0 2024-03-25T18:12:30.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.235+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4670001e80 con 0x7f46781d41b0 2024-03-25T18:12:30.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.235+0000 7f46757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:30.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.235+0000 7f46757fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:30.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.235+0000 7f46757fa640 10 monclient(hunting): dump: 2024-03-25T18:12:30.236 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:30.236 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:30.237 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:30.237 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:30.237 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:30.237 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:30.237 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:30.237 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:30.237 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:30.237 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:30.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1760690723 0 0) 0x7f467813eba0 con 0x7f467806ab80 2024-03-25T18:12:30.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 >> v1:172.21.15.86:6789/0 conn(0x7f467813f990 legacy=0x7f4678137d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:30.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 >> v1:172.21.15.64:6789/0 conn(0x7f46781d41b0 legacy=0x7f4678138450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:30.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f46757fa640 10 monclient: _finish_hunting 0 2024-03-25T18:12:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f46757fa640 1 monclient: found mon.c 2024-03-25T18:12:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f46757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f46757fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:12:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467813d2b0 con 0x7f467806ab80 2024-03-25T18:12:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f46757fa640 10 monclient: _finish_auth 0 2024-03-25T18:12:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f46757fa640 10 monclient: _check_auth_tickets 2024-03-25T18:12:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f46757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f467ec25640 5 monclient: authenticate success, global_id 4358 2024-03-25T18:12:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f467ec25640 10 monclient: _renew_subs 2024-03-25T18:12:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f467ec25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:12:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f467ec25640 1 -- 172.21.15.64:0/3074259887 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467813cfc0 con 0x7f467806ab80 2024-03-25T18:12:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f467ec25640 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f467ec25640 10 monclient: _renew_subs 2024-03-25T18:12:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f467ec25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:12:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f467ec25640 1 -- 172.21.15.64:0/3074259887 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f467813f630 con 0x7f467806ab80 2024-03-25T18:12:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.236+0000 7f467ec25640 1 librados: init done 2024-03-25T18:12:30.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.237+0000 7f467ec25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:30.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.237+0000 7f467ec25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:30.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.237+0000 7f467ec25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:12:30.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.237+0000 7f467ec25640 1 -- 172.21.15.64:0/3074259887 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46780627b0 con 0x7f467806ab80 2024-03-25T18:12:30.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.596+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f466c001c70 con 0x7f467806ab80 2024-03-25T18:12:30.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.596+0000 7f46757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:30.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.596+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f466c003a60 con 0x7f467806ab80 2024-03-25T18:12:30.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.596+0000 7f46757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:30.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.596+0000 7f46757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:30.598 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.596+0000 7f46757fa640 10 monclient: dump: 2024-03-25T18:12:30.598 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:30.598 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:30.598 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:30.598 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:30.598 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:30.598 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:30.598 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:30.598 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:30.598 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:30.598 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:30.598 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.596+0000 7f4676ffd640 4 set_mon_vals no callback set 2024-03-25T18:12:30.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.597+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f466c003c40 con 0x7f467806ab80 2024-03-25T18:12:30.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.597+0000 7f4677fff640 10 monclient: get_auth_request con 0x7f465c06e0f0 auth_method 0 2024-03-25T18:12:30.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.598+0000 7f46757fa640 10 client.4358.objecter ms_handle_connect 0x7f465c06e0f0 2024-03-25T18:12:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.603+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f466c054890 con 0x7f467806ab80 2024-03-25T18:12:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.603+0000 7f46757fa640 10 client.4358.objecter ms_dispatch 0x7f4678000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.603+0000 7f46757fa640 3 client.4358.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.603+0000 7f46757fa640 3 client.4358.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.603+0000 7f46757fa640 20 client.4358.objecter dump_active .. 0 homeless 2024-03-25T18:12:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.603+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f466c0503a0 con 0x7f467806ab80 2024-03-25T18:12:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.603+0000 7f46757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:30.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.603+0000 7f46757fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:30.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.852+0000 7f467ec25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:30.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.853+0000 7f467ec25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:30.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.853+0000 7f467ec25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:12:30.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.853+0000 7f467ec25640 1 -- 172.21.15.64:0/3074259887 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f467813d440 con 0x7f467806ab80 2024-03-25T18:12:30.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.928+0000 7f46757fa640 1 -- 172.21.15.64:0/3074259887 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f466c0522d0 con 0x7f467806ab80 2024-03-25T18:12:30.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.928+0000 7f46757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:30.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.928+0000 7f46757fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:30.929 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:12:30.929 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:30.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.931+0000 7f467ec25640 10 librados: watch_flush enter 2024-03-25T18:12:30.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.931+0000 7f467ec25640 10 librados: watch_flush exit 2024-03-25T18:12:30.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.931+0000 7f467ec25640 20 client.4358.objecter shutdown clearing up homeless session... 2024-03-25T18:12:30.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.931+0000 7f467ec25640 10 client.4358.objecter successfully canceled tick 2024-03-25T18:12:30.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.931+0000 7f467ec25640 1 -- 172.21.15.64:0/3074259887 >> v1:172.21.15.64:6800/2915543638 conn(0x7f465c06e0f0 legacy=0x7f465c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:30.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.931+0000 7f467ec25640 10 monclient: shutdown 2024-03-25T18:12:30.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.931+0000 7f467ec25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:30.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.931+0000 7f467ec25640 1 -- 172.21.15.64:0/3074259887 >> v1:172.21.15.64:6790/0 conn(0x7f467806ab80 legacy=0x7f467813d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:30.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.931+0000 7f467ec25640 1 -- 172.21.15.64:0/3074259887 shutdown_connections 2024-03-25T18:12:30.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.931+0000 7f467ec25640 1 -- 172.21.15.64:0/3074259887 >> 172.21.15.64:0/3074259887 conn(0x7f467806c830 msgr2=0x7f46780b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:30.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.931+0000 7f467ec25640 1 -- 172.21.15.64:0/3074259887 shutdown_connections 2024-03-25T18:12:30.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.931+0000 7f467ec25640 1 -- 172.21.15.64:0/3074259887 wait complete. 2024-03-25T18:12:30.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:30.931+0000 7f467ec25640 1 librados: shutdown 2024-03-25T18:12:30.944 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:31.740 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:03:14.853 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:31.749 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:03:14.862 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:31.783 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:03:14.896 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:31.796 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:03:14.874 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:31.806 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:03:14.884 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:31.837 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:03:14.916 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:31.838 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:03:14.951 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:31.846 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:03:14.959 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:31.870 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:03:14.983 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:37.751 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:37.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.853+0000 7fd13243e640 10 monclient: get_monmap_and_config 2024-03-25T18:12:37.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.853+0000 7fd13243e640 10 monclient: build_initial_monmap 2024-03-25T18:12:37.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.853+0000 7fd13243e640 10 monclient: monmap: 2024-03-25T18:12:37.855 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:12:37.855 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:37.855 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:12:37.855353+0000 2024-03-25T18:12:37.855 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:12:37.855353+0000 2024-03-25T18:12:37.855 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:37.855 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:37.855 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:12:37.855 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:12:37.855 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:12:37.856 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:37.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.854+0000 7fd13243e640 1 Processor -- start 2024-03-25T18:12:37.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.854+0000 7fd13243e640 1 -- start start 2024-03-25T18:12:37.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.854+0000 7fd13243e640 10 monclient: init 2024-03-25T18:12:37.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.854+0000 7fd13243e640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:37.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.854+0000 7fd13243e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:37.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.854+0000 7fd13243e640 10 monclient(hunting): picked mon.noname-b con 0x7fd12c0a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:12:37.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.854+0000 7fd13243e640 10 monclient(hunting): picked mon.noname-a con 0x7fd12c13f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:12:37.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.854+0000 7fd13243e640 10 monclient(hunting): picked mon.noname-c con 0x7fd12c1430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:12:37.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.854+0000 7fd13243e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c142520 con 0x7fd12c13f710 2024-03-25T18:12:37.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.854+0000 7fd13243e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c0b28a0 con 0x7fd12c1430c0 2024-03-25T18:12:37.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.854+0000 7fd13243e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c0b2a30 con 0x7fd12c0a98f0 2024-03-25T18:12:37.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.854+0000 7fd13243e640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:37.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.854+0000 7fd13243e640 10 monclient(hunting): authenticate will time out at 7144.094254s 2024-03-25T18:12:37.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.855+0000 7fd1309db640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd12c1430c0 0x7fd12c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:39938/0 (socket says 172.21.15.64:39938) 2024-03-25T18:12:37.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.855+0000 7fd1309db640 1 -- 172.21.15.64:0/1057492077 learned_addr learned my addr 172.21.15.64:0/1057492077 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:12:37.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.896+0000 7fd12affd640 1 -- 172.21.15.64:0/1057492077 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501627172 0 0) 0x7fd12c0b2a30 con 0x7fd12c0a98f0 2024-03-25T18:12:37.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.896+0000 7fd12affd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:37.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.896+0000 7fd12affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:37.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.896+0000 7fd12affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:37.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.896+0000 7fd12affd640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:12:37.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.896+0000 7fd12affd640 1 -- 172.21.15.64:0/1057492077 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd114003520 con 0x7fd12c0a98f0 2024-03-25T18:12:37.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.914+0000 7fd12affd640 1 -- 172.21.15.64:0/1057492077 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2891537212 0 0) 0x7fd114003520 con 0x7fd12c0a98f0 2024-03-25T18:12:37.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.914+0000 7fd12affd640 1 -- 172.21.15.64:0/1057492077 >> v1:172.21.15.86:6789/0 conn(0x7fd12c1430c0 legacy=0x7fd12c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:37.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.914+0000 7fd12affd640 1 -- 172.21.15.64:0/1057492077 >> v1:172.21.15.64:6789/0 conn(0x7fd12c13f710 legacy=0x7fd12c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:37.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.914+0000 7fd12affd640 10 monclient: _finish_hunting 0 2024-03-25T18:12:37.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.914+0000 7fd12affd640 1 monclient: found mon.noname-b 2024-03-25T18:12:37.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.914+0000 7fd12affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:37.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.914+0000 7fd12affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:12:37.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.914+0000 7fd12affd640 1 -- 172.21.15.64:0/1057492077 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12c0b44c0 con 0x7fd12c0a98f0 2024-03-25T18:12:37.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.914+0000 7fd12affd640 10 monclient: _finish_auth 0 2024-03-25T18:12:37.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.914+0000 7fd12affd640 10 monclient: _check_auth_tickets 2024-03-25T18:12:37.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.914+0000 7fd12affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:37.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.914+0000 7fd12affd640 1 -- 172.21.15.64:0/1057492077 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fd11c001690 con 0x7fd12c0a98f0 2024-03-25T18:12:37.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.914+0000 7fd12affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:37.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.914+0000 7fd12affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:37.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.914+0000 7fd12affd640 10 monclient: dump: 2024-03-25T18:12:37.917 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:37.918 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:37.918 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:37.918 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:37.918 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:37.918 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:37.918 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:37.918 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:37.918 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:37.918 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:37.918 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.914+0000 7fd13243e640 5 monclient: authenticate success, global_id 4361 2024-03-25T18:12:37.918 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.914+0000 7fd13243e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:37.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.925+0000 7fd12affd640 1 -- 172.21.15.64:0/1057492077 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd11c0017f0 con 0x7fd12c0a98f0 2024-03-25T18:12:37.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.925+0000 7fd12affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:37.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.925+0000 7fd12affd640 1 -- 172.21.15.64:0/1057492077 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fd11c001c00 con 0x7fd12c0a98f0 2024-03-25T18:12:37.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.925+0000 7fd13243e640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:37.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.925+0000 7fd13243e640 4 set_mon_vals no callback set 2024-03-25T18:12:37.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.925+0000 7fd12affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:37.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.925+0000 7fd12affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:37.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.925+0000 7fd12affd640 10 monclient: dump: 2024-03-25T18:12:37.928 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:37.928 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:37.928 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:37.928 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:37.928 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:37.928 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:37.928 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:37.928 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:37.928 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:37.928 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:37.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.926+0000 7fd13243e640 10 monclient: shutdown 2024-03-25T18:12:37.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.926+0000 7fd13243e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:37.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.926+0000 7fd13243e640 1 -- 172.21.15.64:0/1057492077 >> v1:172.21.15.64:6790/0 conn(0x7fd12c0a98f0 legacy=0x7fd12c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:37.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.926+0000 7fd13243e640 1 -- 172.21.15.64:0/1057492077 shutdown_connections 2024-03-25T18:12:37.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.926+0000 7fd13243e640 1 -- 172.21.15.64:0/1057492077 >> 172.21.15.64:0/1057492077 conn(0x7fd12c0b7ea0 msgr2=0x7fd12c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:37.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.926+0000 7fd13243e640 1 -- 172.21.15.64:0/1057492077 shutdown_connections 2024-03-25T18:12:37.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.926+0000 7fd13243e640 1 -- 172.21.15.64:0/1057492077 wait complete. 2024-03-25T18:12:37.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.927+0000 7fd13243e640 10 monclient: build_initial_monmap 2024-03-25T18:12:37.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.927+0000 7fd13243e640 10 monclient: monmap: 2024-03-25T18:12:37.930 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:12:37.930 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:37.930 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:12:37.930 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:12:37.931 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:37.931 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:12:37.931 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:12:37.931 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:12:37.931 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:12:37.931 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:37.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.928+0000 7fd13243e640 1 librados: starting msgr at 2024-03-25T18:12:37.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.928+0000 7fd13243e640 1 librados: starting objecter 2024-03-25T18:12:37.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.928+0000 7fd13243e640 1 Processor -- start 2024-03-25T18:12:37.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.928+0000 7fd13243e640 1 -- start start 2024-03-25T18:12:37.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.928+0000 7fd13243e640 1 librados: setting wanted keys 2024-03-25T18:12:37.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.928+0000 7fd13243e640 1 librados: calling monclient init 2024-03-25T18:12:37.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.928+0000 7fd13243e640 10 monclient: init 2024-03-25T18:12:37.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.929+0000 7fd13243e640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:37.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.929+0000 7fd13243e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:37.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.929+0000 7fd13243e640 10 monclient(hunting): picked mon.noname-b con 0x7fd12c0a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:12:37.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.929+0000 7fd13243e640 10 monclient(hunting): picked mon.noname-c con 0x7fd12c13f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:12:37.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.929+0000 7fd13243e640 10 monclient(hunting): picked mon.noname-a con 0x7fd12c1430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:12:37.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.929+0000 7fd13243e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c138a50 con 0x7fd12c1430c0 2024-03-25T18:12:37.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.929+0000 7fd13243e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c0ab9e0 con 0x7fd12c0a98f0 2024-03-25T18:12:37.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.929+0000 7fd13243e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c1d7880 con 0x7fd12c13f710 2024-03-25T18:12:37.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.929+0000 7fd13243e640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:37.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.929+0000 7fd13243e640 10 monclient(hunting): authenticate will time out at 7144.168783s 2024-03-25T18:12:37.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.941+0000 7fd12bfff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd12c0a98f0 0x7fd12c138340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:40000/0 (socket says 172.21.15.64:40000) 2024-03-25T18:12:37.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.941+0000 7fd12bfff640 1 -- 172.21.15.64:0/2156560354 learned_addr learned my addr 172.21.15.64:0/2156560354 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:12:37.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.942+0000 7fd128ff9640 10 client.?.objecter ms_handle_connect 0x7fd12c0a98f0 2024-03-25T18:12:37.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.942+0000 7fd128ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:37.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.944+0000 7fd128ff9640 10 client.?.objecter ms_handle_connect 0x7fd12c1430c0 2024-03-25T18:12:37.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.944+0000 7fd128ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:37.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.948+0000 7fd128ff9640 10 client.?.objecter ms_handle_connect 0x7fd12c13f710 2024-03-25T18:12:37.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.948+0000 7fd128ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:37.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.953+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399993145 0 0) 0x7fd12c0ab9e0 con 0x7fd12c0a98f0 2024-03-25T18:12:37.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.953+0000 7fd128ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:37.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.953+0000 7fd128ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:37.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.953+0000 7fd128ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:37.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.953+0000 7fd128ff9640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:12:37.956 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.953+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd118004eb0 con 0x7fd12c0a98f0 2024-03-25T18:12:37.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.961+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716574956 0 0) 0x7fd12c138a50 con 0x7fd12c1430c0 2024-03-25T18:12:37.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.961+0000 7fd128ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:37.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.961+0000 7fd128ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:37.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.961+0000 7fd128ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:37.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.961+0000 7fd128ff9640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:12:37.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.961+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd12c0ab9e0 con 0x7fd12c1430c0 2024-03-25T18:12:37.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.965+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2043887482 0 0) 0x7fd12c1d7880 con 0x7fd12c13f710 2024-03-25T18:12:37.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.965+0000 7fd128ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:37.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.965+0000 7fd128ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:37.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.965+0000 7fd128ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:37.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.965+0000 7fd128ff9640 10 monclient(hunting): my global_id is 4364 2024-03-25T18:12:37.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.965+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd12c138a50 con 0x7fd12c13f710 2024-03-25T18:12:37.975 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.974+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4238307581 0 0) 0x7fd118004eb0 con 0x7fd12c0a98f0 2024-03-25T18:12:37.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.974+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd12c1d7880 con 0x7fd12c0a98f0 2024-03-25T18:12:37.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.974+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fd11c001690 con 0x7fd12c0a98f0 2024-03-25T18:12:37.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.974+0000 7fd128ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:37.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.974+0000 7fd128ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:37.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.974+0000 7fd128ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:37.976 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:37.976 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:37.976 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:37.976 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:37.976 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:37.976 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:37.976 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:37.976 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:37.976 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:37.976 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:37.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.974+0000 7fd128ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:37.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1707353491 0 0) 0x7fd12c1d7880 con 0x7fd12c0a98f0 2024-03-25T18:12:37.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 >> v1:172.21.15.64:6790/0 conn(0x7fd12c13f710 legacy=0x7fd12c1d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:37.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 >> v1:172.21.15.64:6789/0 conn(0x7fd12c1430c0 legacy=0x7fd12c1e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:37.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd128ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:12:37.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd128ff9640 1 monclient: found mon.b 2024-03-25T18:12:37.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd128ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:37.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd128ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:12:37.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12c1d84b0 con 0x7fd12c0a98f0 2024-03-25T18:12:37.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd128ff9640 10 monclient: _finish_auth 0 2024-03-25T18:12:37.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd128ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:12:37.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd128ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:37.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd13243e640 5 monclient: authenticate success, global_id 4354 2024-03-25T18:12:37.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd13243e640 10 monclient: _renew_subs 2024-03-25T18:12:37.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd13243e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:12:37.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd13243e640 1 -- 172.21.15.64:0/2156560354 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd12c1d51f0 con 0x7fd12c0a98f0 2024-03-25T18:12:37.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd13243e640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:37.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd13243e640 10 monclient: _renew_subs 2024-03-25T18:12:37.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd13243e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:12:37.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd13243e640 1 -- 172.21.15.64:0/2156560354 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd12c138e60 con 0x7fd12c0a98f0 2024-03-25T18:12:37.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.984+0000 7fd13243e640 1 librados: init done 2024-03-25T18:12:37.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.985+0000 7fd13243e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:37.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.985+0000 7fd13243e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:37.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.985+0000 7fd13243e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:12:37.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:37.985+0000 7fd13243e640 1 -- 172.21.15.64:0/2156560354 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd12c1d76e0 con 0x7fd12c0a98f0 2024-03-25T18:12:38.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.272+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd11c0038a0 con 0x7fd12c0a98f0 2024-03-25T18:12:38.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.272+0000 7fd128ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:38.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.272+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fd11c003b50 con 0x7fd12c0a98f0 2024-03-25T18:12:38.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.272+0000 7fd128ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:38.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.272+0000 7fd128ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:38.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.272+0000 7fd128ff9640 10 monclient: dump: 2024-03-25T18:12:38.274 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:38.274 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:38.274 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:38.274 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:38.274 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:38.274 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:38.274 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:38.274 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:38.274 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:38.274 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:38.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.272+0000 7fd12a7fc640 4 set_mon_vals no callback set 2024-03-25T18:12:38.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.272+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fd11c003d30 con 0x7fd12c0a98f0 2024-03-25T18:12:38.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.273+0000 7fd12b7fe640 10 monclient: get_auth_request con 0x7fd11806ddf0 auth_method 0 2024-03-25T18:12:38.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.273+0000 7fd128ff9640 10 client.4354.objecter ms_handle_connect 0x7fd11806ddf0 2024-03-25T18:12:38.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.277+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fd11c054880 con 0x7fd12c0a98f0 2024-03-25T18:12:38.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.277+0000 7fd128ff9640 10 client.4354.objecter ms_dispatch 0x7fd12c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:38.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.278+0000 7fd128ff9640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:38.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.278+0000 7fd128ff9640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:38.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.278+0000 7fd128ff9640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-25T18:12:38.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.278+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd11c050390 con 0x7fd12c0a98f0 2024-03-25T18:12:38.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.278+0000 7fd128ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:38.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.278+0000 7fd128ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:38.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.526+0000 7fd13243e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:38.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.526+0000 7fd13243e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:38.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.526+0000 7fd13243e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:12:38.528 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.526+0000 7fd13243e640 1 -- 172.21.15.64:0/2156560354 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd12c1d7a10 con 0x7fd12c0a98f0 2024-03-25T18:12:38.557 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.556+0000 7fd128ff9640 1 -- 172.21.15.64:0/2156560354 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fd11c0522c0 con 0x7fd12c0a98f0 2024-03-25T18:12:38.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.556+0000 7fd128ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:38.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.556+0000 7fd128ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:38.558 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:12:38.558 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:38.560 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.558+0000 7fd13243e640 10 librados: watch_flush enter 2024-03-25T18:12:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.558+0000 7fd13243e640 10 librados: watch_flush exit 2024-03-25T18:12:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.558+0000 7fd13243e640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-25T18:12:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.558+0000 7fd13243e640 10 client.4354.objecter successfully canceled tick 2024-03-25T18:12:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.558+0000 7fd13243e640 1 -- 172.21.15.64:0/2156560354 >> v1:172.21.15.64:6800/2915543638 conn(0x7fd11806ddf0 legacy=0x7fd1180702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.558+0000 7fd13243e640 10 monclient: shutdown 2024-03-25T18:12:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.558+0000 7fd13243e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.558+0000 7fd13243e640 1 -- 172.21.15.64:0/2156560354 >> v1:172.21.15.86:6789/0 conn(0x7fd12c0a98f0 legacy=0x7fd12c138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.558+0000 7fd13243e640 1 -- 172.21.15.64:0/2156560354 shutdown_connections 2024-03-25T18:12:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.558+0000 7fd13243e640 1 -- 172.21.15.64:0/2156560354 >> 172.21.15.64:0/2156560354 conn(0x7fd12c0b7ea0 msgr2=0x7fd12c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.559+0000 7fd13243e640 1 -- 172.21.15.64:0/2156560354 shutdown_connections 2024-03-25T18:12:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.559+0000 7fd13243e640 1 -- 172.21.15.64:0/2156560354 wait complete. 2024-03-25T18:12:38.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:38.559+0000 7fd13243e640 1 librados: shutdown 2024-03-25T18:12:38.572 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:45.379 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:45.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.496+0000 7f0ffa996640 10 monclient: get_monmap_and_config 2024-03-25T18:12:45.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.496+0000 7f0ffa996640 10 monclient: build_initial_monmap 2024-03-25T18:12:45.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.496+0000 7f0ffa996640 10 monclient: monmap: 2024-03-25T18:12:45.497 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:12:45.497 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:45.497 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:12:45.497165+0000 2024-03-25T18:12:45.497 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:12:45.497165+0000 2024-03-25T18:12:45.497 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:45.497 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:45.497 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:12:45.497 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:12:45.497 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:12:45.498 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:45.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.497+0000 7f0ffa996640 1 Processor -- start 2024-03-25T18:12:45.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.497+0000 7f0ffa996640 1 -- start start 2024-03-25T18:12:45.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.497+0000 7f0ffa996640 10 monclient: init 2024-03-25T18:12:45.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.498+0000 7f0ffa996640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:45.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.498+0000 7f0ffa996640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:45.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.498+0000 7f0ffa996640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff40a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:12:45.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.498+0000 7f0ffa996640 10 monclient(hunting): picked mon.noname-a con 0x7f0ff413f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:12:45.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.498+0000 7f0ffa996640 10 monclient(hunting): picked mon.noname-c con 0x7f0ff41430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:12:45.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.498+0000 7f0ffa996640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff4142520 con 0x7f0ff413f710 2024-03-25T18:12:45.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.498+0000 7f0ffa996640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff40b28a0 con 0x7f0ff41430c0 2024-03-25T18:12:45.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.498+0000 7f0ffa996640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff40b2a30 con 0x7f0ff40a98f0 2024-03-25T18:12:45.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.498+0000 7f0ffa996640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:45.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.498+0000 7f0ffa996640 10 monclient(hunting): authenticate will time out at 7151.736789s 2024-03-25T18:12:45.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.507+0000 7f0ff8f33640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f0ff41430c0 0x7f0ff414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:44498/0 (socket says 172.21.15.64:44498) 2024-03-25T18:12:45.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.507+0000 7f0ff8f33640 1 -- 172.21.15.64:0/3563543215 learned_addr learned my addr 172.21.15.64:0/3563543215 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:12:45.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.516+0000 7f0ff37fe640 1 -- 172.21.15.64:0/3563543215 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450119403 0 0) 0x7f0ff40b28a0 con 0x7f0ff41430c0 2024-03-25T18:12:45.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.516+0000 7f0ff37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:45.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.516+0000 7f0ff37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:45.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.516+0000 7f0ff37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:45.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.516+0000 7f0ff37fe640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:12:45.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.516+0000 7f0ff37fe640 1 -- 172.21.15.64:0/3563543215 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fe0003520 con 0x7f0ff41430c0 2024-03-25T18:12:45.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.525+0000 7f0ff37fe640 1 -- 172.21.15.64:0/3563543215 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176107250 0 0) 0x7f0ff4142520 con 0x7f0ff413f710 2024-03-25T18:12:45.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.525+0000 7f0ff37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:45.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.525+0000 7f0ff37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:45.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.525+0000 7f0ff37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:45.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.525+0000 7f0ff37fe640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:12:45.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.525+0000 7f0ff37fe640 1 -- 172.21.15.64:0/3563543215 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff40b28a0 con 0x7f0ff413f710 2024-03-25T18:12:45.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.535+0000 7f0ff37fe640 1 -- 172.21.15.64:0/3563543215 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2544442481 0 0) 0x7f0fe0003520 con 0x7f0ff41430c0 2024-03-25T18:12:45.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.535+0000 7f0ff37fe640 1 -- 172.21.15.64:0/3563543215 >> v1:172.21.15.64:6790/0 conn(0x7f0ff40a98f0 legacy=0x7f0ff40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:45.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.535+0000 7f0ff37fe640 1 -- 172.21.15.64:0/3563543215 >> v1:172.21.15.64:6789/0 conn(0x7f0ff413f710 legacy=0x7f0ff41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:45.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.535+0000 7f0ff37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:12:45.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.535+0000 7f0ff37fe640 1 monclient: found mon.noname-c 2024-03-25T18:12:45.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.535+0000 7f0ff37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:45.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.535+0000 7f0ff37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:12:45.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.535+0000 7f0ff37fe640 1 -- 172.21.15.64:0/3563543215 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff40b44c0 con 0x7f0ff41430c0 2024-03-25T18:12:45.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.535+0000 7f0ff37fe640 10 monclient: _finish_auth 0 2024-03-25T18:12:45.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.535+0000 7f0ff37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:12:45.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.535+0000 7f0ff37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:45.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.535+0000 7f0ff37fe640 1 -- 172.21.15.64:0/3563543215 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0fec001a10 con 0x7f0ff41430c0 2024-03-25T18:12:45.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.535+0000 7f0ffa996640 5 monclient: authenticate success, global_id 4357 2024-03-25T18:12:45.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.535+0000 7f0ffa996640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:45.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.535+0000 7f0ff37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:45.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.535+0000 7f0ff37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:45.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.535+0000 7f0ff37fe640 10 monclient: dump: 2024-03-25T18:12:45.537 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:45.537 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:45.537 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:45.537 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:45.537 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:45.537 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:45.538 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:45.538 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:45.538 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:45.538 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:45.538 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.535+0000 7f0ffa996640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:45.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.558+0000 7f0ff37fe640 1 -- 172.21.15.64:0/3563543215 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fec001b70 con 0x7f0ff41430c0 2024-03-25T18:12:45.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.558+0000 7f0ff37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:45.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.558+0000 7f0ff37fe640 1 -- 172.21.15.64:0/3563543215 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0fec001f80 con 0x7f0ff41430c0 2024-03-25T18:12:45.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.558+0000 7f0ff37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:45.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.558+0000 7f0ff37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:45.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.558+0000 7f0ff37fe640 10 monclient: dump: 2024-03-25T18:12:45.559 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:45.559 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:45.559 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:45.559 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:45.559 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:45.560 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:45.560 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:45.560 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:45.560 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:45.560 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:45.560 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.558+0000 7f0ffa996640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:45.560 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.558+0000 7f0ffa996640 4 set_mon_vals no callback set 2024-03-25T18:12:45.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.559+0000 7f0ffa996640 10 monclient: shutdown 2024-03-25T18:12:45.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.559+0000 7f0ffa996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:45.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.559+0000 7f0ffa996640 1 -- 172.21.15.64:0/3563543215 >> v1:172.21.15.86:6789/0 conn(0x7f0ff41430c0 legacy=0x7f0ff414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:45.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.559+0000 7f0ffa996640 1 -- 172.21.15.64:0/3563543215 shutdown_connections 2024-03-25T18:12:45.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.559+0000 7f0ffa996640 1 -- 172.21.15.64:0/3563543215 >> 172.21.15.64:0/3563543215 conn(0x7f0ff40b7ea0 msgr2=0x7f0ff40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:45.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.559+0000 7f0ffa996640 1 -- 172.21.15.64:0/3563543215 shutdown_connections 2024-03-25T18:12:45.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.559+0000 7f0ffa996640 1 -- 172.21.15.64:0/3563543215 wait complete. 2024-03-25T18:12:45.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.560+0000 7f0ffa996640 10 monclient: build_initial_monmap 2024-03-25T18:12:45.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.560+0000 7f0ffa996640 10 monclient: monmap: 2024-03-25T18:12:45.562 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:12:45.562 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:45.562 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:12:45.562 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:12:45.562 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:45.562 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:12:45.562 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:12:45.562 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:12:45.562 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:12:45.562 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:45.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.560+0000 7f0ffa996640 1 librados: starting msgr at 2024-03-25T18:12:45.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.560+0000 7f0ffa996640 1 librados: starting objecter 2024-03-25T18:12:45.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.560+0000 7f0ffa996640 1 Processor -- start 2024-03-25T18:12:45.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.560+0000 7f0ffa996640 1 -- start start 2024-03-25T18:12:45.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.560+0000 7f0ffa996640 1 librados: setting wanted keys 2024-03-25T18:12:45.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.560+0000 7f0ffa996640 1 librados: calling monclient init 2024-03-25T18:12:45.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.560+0000 7f0ffa996640 10 monclient: init 2024-03-25T18:12:45.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.561+0000 7f0ffa996640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:45.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.561+0000 7f0ffa996640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:45.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.561+0000 7f0ffa996640 10 monclient(hunting): picked mon.noname-b con 0x7f0ff40a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:12:45.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.561+0000 7f0ffa996640 10 monclient(hunting): picked mon.noname-a con 0x7f0ff413f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:12:45.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.561+0000 7f0ffa996640 10 monclient(hunting): picked mon.noname-c con 0x7f0ff41430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:12:45.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.561+0000 7f0ffa996640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff4135230 con 0x7f0ff413f710 2024-03-25T18:12:45.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.561+0000 7f0ffa996640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff4135da0 con 0x7f0ff40a98f0 2024-03-25T18:12:45.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.561+0000 7f0ffa996640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff4135f30 con 0x7f0ff41430c0 2024-03-25T18:12:45.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.561+0000 7f0ffa996640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:45.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.561+0000 7f0ffa996640 10 monclient(hunting): authenticate will time out at 7151.800027s 2024-03-25T18:12:45.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.567+0000 7f0ff8f33640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f0ff41430c0 0x7f0ff41db560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:40584/0 (socket says 172.21.15.64:40584) 2024-03-25T18:12:45.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.567+0000 7f0ff8f33640 1 -- 172.21.15.64:0/2225468252 learned_addr learned my addr 172.21.15.64:0/2225468252 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:12:45.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.571+0000 7f0ff17fa640 10 client.?.objecter ms_handle_connect 0x7f0ff41430c0 2024-03-25T18:12:45.572 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.571+0000 7f0ff17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:45.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.585+0000 7f0ff17fa640 10 client.?.objecter ms_handle_connect 0x7f0ff40a98f0 2024-03-25T18:12:45.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.585+0000 7f0ff17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:45.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.590+0000 7f0ff17fa640 10 client.?.objecter ms_handle_connect 0x7f0ff413f710 2024-03-25T18:12:45.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.590+0000 7f0ff17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:45.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.591+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499832733 0 0) 0x7f0ff4135f30 con 0x7f0ff41430c0 2024-03-25T18:12:45.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.591+0000 7f0ff17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:45.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.591+0000 7f0ff17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:45.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.591+0000 7f0ff17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:45.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.591+0000 7f0ff17fa640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:12:45.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.591+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd4004eb0 con 0x7f0ff41430c0 2024-03-25T18:12:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.595+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668753403 0 0) 0x7f0ff4135da0 con 0x7f0ff40a98f0 2024-03-25T18:12:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.595+0000 7f0ff17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.595+0000 7f0ff17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.595+0000 7f0ff17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.595+0000 7f0ff17fa640 10 monclient(hunting): my global_id is 4360 2024-03-25T18:12:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.595+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff4135f30 con 0x7f0ff40a98f0 2024-03-25T18:12:45.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.605+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4271335461 0 0) 0x7f0ff4135f30 con 0x7f0ff40a98f0 2024-03-25T18:12:45.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.605+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ff4135da0 con 0x7f0ff40a98f0 2024-03-25T18:12:45.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.605+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0fe40018b0 con 0x7f0ff40a98f0 2024-03-25T18:12:45.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.605+0000 7f0ff17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:45.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.605+0000 7f0ff17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:45.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.605+0000 7f0ff17fa640 10 monclient(hunting): dump: 2024-03-25T18:12:45.606 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:45.606 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:45.606 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:45.607 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:45.607 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:45.607 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:45.607 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:45.607 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:45.607 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:45.607 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:45.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.605+0000 7f0ff17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:45.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.606+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 586366760 0 0) 0x7f0fd4004eb0 con 0x7f0ff41430c0 2024-03-25T18:12:45.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.606+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ff4135f30 con 0x7f0ff41430c0 2024-03-25T18:12:45.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.606+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0fec001ec0 con 0x7f0ff41430c0 2024-03-25T18:12:45.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.606+0000 7f0ff17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:45.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.606+0000 7f0ff17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:45.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.606+0000 7f0ff17fa640 10 monclient(hunting): dump: 2024-03-25T18:12:45.608 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:45.608 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:45.608 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:45.608 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:45.608 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:45.608 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:45.608 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:45.608 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:45.608 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:45.609 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:45.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.613+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3796242036 0 0) 0x7f0ff4135230 con 0x7f0ff413f710 2024-03-25T18:12:45.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.613+0000 7f0ff17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:45.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.613+0000 7f0ff17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:45.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.613+0000 7f0ff17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:45.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.613+0000 7f0ff17fa640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:12:45.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.613+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd4004eb0 con 0x7f0ff413f710 2024-03-25T18:12:45.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1213283318 0 0) 0x7f0ff4135da0 con 0x7f0ff40a98f0 2024-03-25T18:12:45.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 >> v1:172.21.15.64:6790/0 conn(0x7f0ff41430c0 legacy=0x7f0ff41db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:45.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 >> v1:172.21.15.64:6789/0 conn(0x7f0ff413f710 legacy=0x7f0ff41dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:45.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ff17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:12:45.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ff17fa640 1 monclient: found mon.b 2024-03-25T18:12:45.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ff17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:45.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ff17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:12:45.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff41360c0 con 0x7f0ff40a98f0 2024-03-25T18:12:45.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ff17fa640 10 monclient: _finish_auth 0 2024-03-25T18:12:45.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ff17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:12:45.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ff17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:45.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ffa996640 5 monclient: authenticate success, global_id 4360 2024-03-25T18:12:45.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ffa996640 10 monclient: _renew_subs 2024-03-25T18:12:45.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ffa996640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:12:45.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ffa996640 1 -- 172.21.15.64:0/2225468252 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff41d5000 con 0x7f0ff40a98f0 2024-03-25T18:12:45.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ffa996640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:45.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ffa996640 10 monclient: _renew_subs 2024-03-25T18:12:45.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ffa996640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:12:45.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ffa996640 1 -- 172.21.15.64:0/2225468252 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff41d41e0 con 0x7f0ff40a98f0 2024-03-25T18:12:45.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.617+0000 7f0ffa996640 1 librados: init done 2024-03-25T18:12:45.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.618+0000 7f0ffa996640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:45.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.618+0000 7f0ffa996640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:45.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.618+0000 7f0ffa996640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:12:45.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.618+0000 7f0ffa996640 1 -- 172.21.15.64:0/2225468252 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff4135090 con 0x7f0ff40a98f0 2024-03-25T18:12:45.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.888+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fe4003610 con 0x7f0ff40a98f0 2024-03-25T18:12:45.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.888+0000 7f0ff17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:45.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.888+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0fe40038c0 con 0x7f0ff40a98f0 2024-03-25T18:12:45.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.888+0000 7f0ff17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:45.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.888+0000 7f0ff17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:45.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.888+0000 7f0ff17fa640 10 monclient: dump: 2024-03-25T18:12:45.889 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:45.890 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:45.890 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:45.890 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:45.890 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:45.890 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:45.890 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:45.890 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:45.890 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:45.890 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:45.890 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.888+0000 7f0ff2ffd640 4 set_mon_vals no callback set 2024-03-25T18:12:45.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.889+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f0fe4003ae0 con 0x7f0ff40a98f0 2024-03-25T18:12:45.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.889+0000 7f0febfff640 10 monclient: get_auth_request con 0x7f0fd406df50 auth_method 0 2024-03-25T18:12:45.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.889+0000 7f0ff17fa640 10 client.4360.objecter ms_handle_connect 0x7f0fd406df50 2024-03-25T18:12:45.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.893+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f0fe40533b0 con 0x7f0ff40a98f0 2024-03-25T18:12:45.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.893+0000 7f0ff17fa640 10 client.4360.objecter ms_dispatch 0x7f0ff4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:45.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.893+0000 7f0ff17fa640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:45.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.893+0000 7f0ff17fa640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:45.894 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.893+0000 7f0ff17fa640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-25T18:12:45.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.894+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0fe404eec0 con 0x7f0ff40a98f0 2024-03-25T18:12:45.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.894+0000 7f0ff17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:45.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:45.894+0000 7f0ff17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:46.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.140+0000 7f0ffa996640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:46.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.140+0000 7f0ffa996640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:46.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.140+0000 7f0ffa996640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:12:46.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.140+0000 7f0ffa996640 1 -- 172.21.15.64:0/2225468252 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ff413b700 con 0x7f0ff40a98f0 2024-03-25T18:12:46.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.161+0000 7f0ff17fa640 1 -- 172.21.15.64:0/2225468252 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f0fe4050df0 con 0x7f0ff40a98f0 2024-03-25T18:12:46.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.161+0000 7f0ff17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:46.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.161+0000 7f0ff17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:46.162 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:12:46.162 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:46.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.164+0000 7f0ffa996640 10 librados: watch_flush enter 2024-03-25T18:12:46.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.164+0000 7f0ffa996640 10 librados: watch_flush exit 2024-03-25T18:12:46.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.164+0000 7f0ffa996640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-25T18:12:46.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.164+0000 7f0ffa996640 10 client.4360.objecter successfully canceled tick 2024-03-25T18:12:46.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.164+0000 7f0ffa996640 1 -- 172.21.15.64:0/2225468252 >> v1:172.21.15.64:6800/2915543638 conn(0x7f0fd406df50 legacy=0x7f0fd4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:46.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.164+0000 7f0ffa996640 10 monclient: shutdown 2024-03-25T18:12:46.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.164+0000 7f0ffa996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:46.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.164+0000 7f0ffa996640 1 -- 172.21.15.64:0/2225468252 >> v1:172.21.15.86:6789/0 conn(0x7f0ff40a98f0 legacy=0x7f0ff41d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:46.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.164+0000 7f0ffa996640 1 -- 172.21.15.64:0/2225468252 shutdown_connections 2024-03-25T18:12:46.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.164+0000 7f0ffa996640 1 -- 172.21.15.64:0/2225468252 >> 172.21.15.64:0/2225468252 conn(0x7f0ff40b7ea0 msgr2=0x7f0ff414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:46.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.164+0000 7f0ffa996640 1 -- 172.21.15.64:0/2225468252 shutdown_connections 2024-03-25T18:12:46.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.164+0000 7f0ffa996640 1 -- 172.21.15.64:0/2225468252 wait complete. 2024-03-25T18:12:46.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:46.164+0000 7f0ffa996640 1 librados: shutdown 2024-03-25T18:12:46.178 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:51.810 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:03:34.922 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:51.813 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:03:34.926 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:51.840 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:03:34.953 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:51.872 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:03:34.951 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:51.879 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:03:34.991 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:51.879 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:03:34.958 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:51.891 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:03:35.004 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:51.901 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:03:34.980 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:51.919 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:03:35.032 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:52.984 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:53.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.082+0000 7fabcd9af640 10 monclient: get_monmap_and_config 2024-03-25T18:12:53.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.082+0000 7fabcd9af640 10 monclient: build_initial_monmap 2024-03-25T18:12:53.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.082+0000 7fabcd9af640 10 monclient: monmap: 2024-03-25T18:12:53.083 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:12:53.084 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:53.084 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:12:53.083564+0000 2024-03-25T18:12:53.084 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:12:53.083564+0000 2024-03-25T18:12:53.084 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:53.084 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:53.084 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:12:53.084 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:12:53.084 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:12:53.084 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:53.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.083+0000 7fabcd9af640 1 Processor -- start 2024-03-25T18:12:53.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.083+0000 7fabcd9af640 1 -- start start 2024-03-25T18:12:53.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.083+0000 7fabcd9af640 10 monclient: init 2024-03-25T18:12:53.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.083+0000 7fabcd9af640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:53.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.083+0000 7fabcd9af640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:53.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.083+0000 7fabcd9af640 10 monclient(hunting): picked mon.noname-c con 0x7fabc80a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:12:53.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.083+0000 7fabcd9af640 10 monclient(hunting): picked mon.noname-b con 0x7fabc813f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:12:53.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.084+0000 7fabcd9af640 10 monclient(hunting): picked mon.noname-a con 0x7fabc81430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:12:53.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.084+0000 7fabcd9af640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc8142520 con 0x7fabc81430c0 2024-03-25T18:12:53.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.084+0000 7fabcd9af640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc80b28a0 con 0x7fabc80a98f0 2024-03-25T18:12:53.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.084+0000 7fabcd9af640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc80b2a30 con 0x7fabc813f710 2024-03-25T18:12:53.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.084+0000 7fabcd9af640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:53.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.084+0000 7fabcd9af640 10 monclient(hunting): authenticate will time out at 7159.322523s 2024-03-25T18:12:53.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.084+0000 7fabc6ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fabc80a98f0 0x7fabc80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:33888/0 (socket says 172.21.15.64:33888) 2024-03-25T18:12:53.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.084+0000 7fabc6ffd640 1 -- 172.21.15.64:0/2212987213 learned_addr learned my addr 172.21.15.64:0/2212987213 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:12:53.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.113+0000 7fabc5ffb640 1 -- 172.21.15.64:0/2212987213 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824408607 0 0) 0x7fabc80b28a0 con 0x7fabc80a98f0 2024-03-25T18:12:53.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.113+0000 7fabc5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:53.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.113+0000 7fabc5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:53.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.113+0000 7fabc5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:53.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.113+0000 7fabc5ffb640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:12:53.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.113+0000 7fabc5ffb640 1 -- 172.21.15.64:0/2212987213 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb0003520 con 0x7fabc80a98f0 2024-03-25T18:12:53.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.121+0000 7fabc5ffb640 1 -- 172.21.15.64:0/2212987213 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821751903 0 0) 0x7fabc80b2a30 con 0x7fabc813f710 2024-03-25T18:12:53.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.121+0000 7fabc5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:53.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.121+0000 7fabc5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:53.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.121+0000 7fabc5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:53.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.121+0000 7fabc5ffb640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:12:53.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.121+0000 7fabc5ffb640 1 -- 172.21.15.64:0/2212987213 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc80b28a0 con 0x7fabc813f710 2024-03-25T18:12:53.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.121+0000 7fabc5ffb640 1 -- 172.21.15.64:0/2212987213 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2081768837 0 0) 0x7fabb0003520 con 0x7fabc80a98f0 2024-03-25T18:12:53.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.121+0000 7fabc5ffb640 1 -- 172.21.15.64:0/2212987213 >> v1:172.21.15.64:6790/0 conn(0x7fabc813f710 legacy=0x7fabc81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:53.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.121+0000 7fabc5ffb640 1 -- 172.21.15.64:0/2212987213 >> v1:172.21.15.64:6789/0 conn(0x7fabc81430c0 legacy=0x7fabc814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:53.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.122+0000 7fabc5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:12:53.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.122+0000 7fabc5ffb640 1 monclient: found mon.noname-c 2024-03-25T18:12:53.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.122+0000 7fabc5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:53.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.122+0000 7fabc5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:12:53.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.122+0000 7fabc5ffb640 1 -- 172.21.15.64:0/2212987213 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc80b44c0 con 0x7fabc80a98f0 2024-03-25T18:12:53.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.122+0000 7fabc5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:12:53.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.122+0000 7fabc5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:12:53.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.122+0000 7fabc5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:53.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.122+0000 7fabc5ffb640 1 -- 172.21.15.64:0/2212987213 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fabb8001690 con 0x7fabc80a98f0 2024-03-25T18:12:53.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.122+0000 7fabc5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:53.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.122+0000 7fabc5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:53.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.122+0000 7fabc5ffb640 10 monclient: dump: 2024-03-25T18:12:53.123 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:53.124 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:53.124 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:53.124 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:53.124 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:53.124 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:53.124 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:53.124 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:53.124 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:53.124 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:53.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.122+0000 7fabcd9af640 5 monclient: authenticate success, global_id 4363 2024-03-25T18:12:53.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.122+0000 7fabcd9af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:53.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.142+0000 7fabc5ffb640 1 -- 172.21.15.64:0/2212987213 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabb80017f0 con 0x7fabc80a98f0 2024-03-25T18:12:53.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.142+0000 7fabc5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:53.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.142+0000 7fabc5ffb640 1 -- 172.21.15.64:0/2212987213 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fabb8001c00 con 0x7fabc80a98f0 2024-03-25T18:12:53.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.142+0000 7fabc5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:53.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.142+0000 7fabc5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:53.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.142+0000 7fabc5ffb640 10 monclient: dump: 2024-03-25T18:12:53.143 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:53.143 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:53.143 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:53.143 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:53.143 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:53.143 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:53.143 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:53.144 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:53.144 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:53.144 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:53.144 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.142+0000 7fabcd9af640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:53.144 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.142+0000 7fabcd9af640 4 set_mon_vals no callback set 2024-03-25T18:12:53.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.143+0000 7fabcd9af640 10 monclient: shutdown 2024-03-25T18:12:53.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.143+0000 7fabcd9af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:53.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.143+0000 7fabcd9af640 1 -- 172.21.15.64:0/2212987213 >> v1:172.21.15.86:6789/0 conn(0x7fabc80a98f0 legacy=0x7fabc80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:53.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.143+0000 7fabcd9af640 1 -- 172.21.15.64:0/2212987213 shutdown_connections 2024-03-25T18:12:53.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.143+0000 7fabcd9af640 1 -- 172.21.15.64:0/2212987213 >> 172.21.15.64:0/2212987213 conn(0x7fabc80b7ea0 msgr2=0x7fabc80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:53.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.143+0000 7fabcd9af640 1 -- 172.21.15.64:0/2212987213 shutdown_connections 2024-03-25T18:12:53.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.143+0000 7fabcd9af640 1 -- 172.21.15.64:0/2212987213 wait complete. 2024-03-25T18:12:53.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.144+0000 7fabcd9af640 10 monclient: build_initial_monmap 2024-03-25T18:12:53.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.144+0000 7fabcd9af640 10 monclient: monmap: 2024-03-25T18:12:53.146 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:12:53.146 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:53.146 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:12:53.146 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:12:53.146 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:53.146 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:12:53.146 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:12:53.146 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:12:53.146 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:12:53.146 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:53.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.144+0000 7fabcd9af640 1 librados: starting msgr at 2024-03-25T18:12:53.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.144+0000 7fabcd9af640 1 librados: starting objecter 2024-03-25T18:12:53.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.144+0000 7fabcd9af640 1 Processor -- start 2024-03-25T18:12:53.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.144+0000 7fabcd9af640 1 -- start start 2024-03-25T18:12:53.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.145+0000 7fabcd9af640 1 librados: setting wanted keys 2024-03-25T18:12:53.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.145+0000 7fabcd9af640 1 librados: calling monclient init 2024-03-25T18:12:53.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.145+0000 7fabcd9af640 10 monclient: init 2024-03-25T18:12:53.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.145+0000 7fabcd9af640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:53.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.145+0000 7fabcd9af640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:53.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.145+0000 7fabcd9af640 10 monclient(hunting): picked mon.noname-a con 0x7fabc80a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:12:53.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.145+0000 7fabcd9af640 10 monclient(hunting): picked mon.noname-b con 0x7fabc813f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:12:53.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.145+0000 7fabcd9af640 10 monclient(hunting): picked mon.noname-c con 0x7fabc81430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:12:53.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.145+0000 7fabcd9af640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc8135730 con 0x7fabc80a98f0 2024-03-25T18:12:53.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.145+0000 7fabcd9af640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc81358c0 con 0x7fabc813f710 2024-03-25T18:12:53.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.145+0000 7fabcd9af640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc8135a50 con 0x7fabc81430c0 2024-03-25T18:12:53.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.145+0000 7fabcd9af640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:53.148 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.145+0000 7fabcd9af640 10 monclient(hunting): authenticate will time out at 7159.384318s 2024-03-25T18:12:53.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.163+0000 7fabc77fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fabc81430c0 0x7fabc81d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:49970/0 (socket says 172.21.15.64:49970) 2024-03-25T18:12:53.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.163+0000 7fabc77fe640 1 -- 172.21.15.64:0/3251030650 learned_addr learned my addr 172.21.15.64:0/3251030650 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:12:53.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.166+0000 7fabaffff640 10 client.?.objecter ms_handle_connect 0x7fabc81430c0 2024-03-25T18:12:53.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.166+0000 7fabaffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:53.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.172+0000 7fabaffff640 10 client.?.objecter ms_handle_connect 0x7fabc80a98f0 2024-03-25T18:12:53.172 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.172+0000 7fabaffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:53.184 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.183+0000 7fabaffff640 10 client.?.objecter ms_handle_connect 0x7fabc813f710 2024-03-25T18:12:53.184 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.183+0000 7fabaffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:53.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.185+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833985324 0 0) 0x7fabc8135a50 con 0x7fabc81430c0 2024-03-25T18:12:53.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.185+0000 7fabaffff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:53.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.185+0000 7fabaffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:53.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.185+0000 7fabaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:53.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.185+0000 7fabaffff640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:12:53.186 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.185+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb4004eb0 con 0x7fabc81430c0 2024-03-25T18:12:53.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.190+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364548440 0 0) 0x7fabc8135730 con 0x7fabc80a98f0 2024-03-25T18:12:53.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.190+0000 7fabaffff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:53.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.190+0000 7fabaffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:53.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.190+0000 7fabaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:53.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.190+0000 7fabaffff640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:12:53.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.190+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc8135a50 con 0x7fabc80a98f0 2024-03-25T18:12:53.196 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.196+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886765726 0 0) 0x7fabc81358c0 con 0x7fabc813f710 2024-03-25T18:12:53.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.196+0000 7fabaffff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:53.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.196+0000 7fabaffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:53.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.196+0000 7fabaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:53.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.196+0000 7fabaffff640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:12:53.197 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.196+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc8135730 con 0x7fabc813f710 2024-03-25T18:12:53.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.197+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2280539286 0 0) 0x7fabb4004eb0 con 0x7fabc81430c0 2024-03-25T18:12:53.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.198+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabc81358c0 con 0x7fabc81430c0 2024-03-25T18:12:53.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.198+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fabc0001c30 con 0x7fabc81430c0 2024-03-25T18:12:53.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.198+0000 7fabaffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:53.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.198+0000 7fabaffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:53.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.198+0000 7fabaffff640 10 monclient(hunting): dump: 2024-03-25T18:12:53.199 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:53.199 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:53.199 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:53.199 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:53.199 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:53.199 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:53.199 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:53.200 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:53.200 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:53.200 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:53.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.198+0000 7fabaffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:53.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.205+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3263044659 0 0) 0x7fabc8135a50 con 0x7fabc80a98f0 2024-03-25T18:12:53.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.205+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabb4004eb0 con 0x7fabc80a98f0 2024-03-25T18:12:53.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.205+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fabb8001b40 con 0x7fabc80a98f0 2024-03-25T18:12:53.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.205+0000 7fabaffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:53.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.205+0000 7fabaffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:53.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.205+0000 7fabaffff640 10 monclient(hunting): dump: 2024-03-25T18:12:53.206 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:53.206 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:53.206 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:53.206 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:53.206 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:53.206 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:53.206 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:53.206 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:53.206 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:53.206 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:53.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3929643858 0 0) 0x7fabc81358c0 con 0x7fabc81430c0 2024-03-25T18:12:53.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 >> v1:172.21.15.86:6789/0 conn(0x7fabc813f710 legacy=0x7fabc81d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:53.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 >> v1:172.21.15.64:6789/0 conn(0x7fabc80a98f0 legacy=0x7fabc81dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:53.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabaffff640 10 monclient: _finish_hunting 0 2024-03-25T18:12:53.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabaffff640 1 monclient: found mon.c 2024-03-25T18:12:53.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:53.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabaffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:12:53.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc81d5000 con 0x7fabc81430c0 2024-03-25T18:12:53.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabaffff640 10 monclient: _finish_auth 0 2024-03-25T18:12:53.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabaffff640 10 monclient: _check_auth_tickets 2024-03-25T18:12:53.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabaffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:53.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabcd9af640 5 monclient: authenticate success, global_id 4376 2024-03-25T18:12:53.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabcd9af640 10 monclient: _renew_subs 2024-03-25T18:12:53.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabcd9af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:12:53.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabcd9af640 1 -- 172.21.15.64:0/3251030650 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc81d52b0 con 0x7fabc81430c0 2024-03-25T18:12:53.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabcd9af640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:53.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabcd9af640 10 monclient: _renew_subs 2024-03-25T18:12:53.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabcd9af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:12:53.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabcd9af640 1 -- 172.21.15.64:0/3251030650 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc813c900 con 0x7fabc81430c0 2024-03-25T18:12:53.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.206+0000 7fabcd9af640 1 librados: init done 2024-03-25T18:12:53.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.207+0000 7fabcd9af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:53.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.207+0000 7fabcd9af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:53.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.207+0000 7fabcd9af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:12:53.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.207+0000 7fabcd9af640 1 -- 172.21.15.64:0/3251030650 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc81371c0 con 0x7fabc81430c0 2024-03-25T18:12:53.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.423+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabc0003990 con 0x7fabc81430c0 2024-03-25T18:12:53.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.423+0000 7fabaffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:53.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.423+0000 7fabc57fa640 4 set_mon_vals no callback set 2024-03-25T18:12:53.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.501+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fabc0003da0 con 0x7fabc81430c0 2024-03-25T18:12:53.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.501+0000 7fabaffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:53.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.501+0000 7fabaffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:53.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.501+0000 7fabaffff640 10 monclient: dump: 2024-03-25T18:12:53.503 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:12:53.503 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:12:53.503 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:53.503 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:12:53.503 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:53.503 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:12:53.503 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:12:53.503 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:12:53.503 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:12:53.503 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:12:53.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.502+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fabc0003f80 con 0x7fabc81430c0 2024-03-25T18:12:53.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.503+0000 7fabc6ffd640 10 monclient: get_auth_request con 0x7fabb406df20 auth_method 0 2024-03-25T18:12:53.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.503+0000 7fabaffff640 10 client.4376.objecter ms_handle_connect 0x7fabb406df20 2024-03-25T18:12:53.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.509+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fabc0054ad0 con 0x7fabc81430c0 2024-03-25T18:12:53.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.509+0000 7fabaffff640 10 client.4376.objecter ms_dispatch 0x7fabc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:53.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.509+0000 7fabaffff640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:53.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.509+0000 7fabaffff640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:53.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.509+0000 7fabaffff640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-25T18:12:53.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.509+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fabc00505e0 con 0x7fabc81430c0 2024-03-25T18:12:53.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.509+0000 7fabaffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:53.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.509+0000 7fabaffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:53.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.761+0000 7fabcd9af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:53.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.761+0000 7fabcd9af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:53.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.761+0000 7fabcd9af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:12:53.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.761+0000 7fabcd9af640 1 -- 172.21.15.64:0/3251030650 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabc80627b0 con 0x7fabc81430c0 2024-03-25T18:12:53.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.788+0000 7fabaffff640 1 -- 172.21.15.64:0/3251030650 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fabc0052510 con 0x7fabc81430c0 2024-03-25T18:12:53.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.788+0000 7fabaffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:53.790 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.788+0000 7fabaffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:53.790 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:12:53.790 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:53.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.791+0000 7fabcd9af640 10 librados: watch_flush enter 2024-03-25T18:12:53.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.791+0000 7fabcd9af640 10 librados: watch_flush exit 2024-03-25T18:12:53.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.791+0000 7fabcd9af640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-25T18:12:53.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.791+0000 7fabcd9af640 10 client.4376.objecter successfully canceled tick 2024-03-25T18:12:53.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.791+0000 7fabcd9af640 1 -- 172.21.15.64:0/3251030650 >> v1:172.21.15.64:6800/2915543638 conn(0x7fabb406df20 legacy=0x7fabb40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:53.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.792+0000 7fabcd9af640 10 monclient: shutdown 2024-03-25T18:12:53.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.792+0000 7fabcd9af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:53.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.792+0000 7fabcd9af640 1 -- 172.21.15.64:0/3251030650 >> v1:172.21.15.64:6790/0 conn(0x7fabc81430c0 legacy=0x7fabc81d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:53.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.792+0000 7fabcd9af640 1 -- 172.21.15.64:0/3251030650 shutdown_connections 2024-03-25T18:12:53.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.792+0000 7fabcd9af640 1 -- 172.21.15.64:0/3251030650 >> 172.21.15.64:0/3251030650 conn(0x7fabc80b7ea0 msgr2=0x7fabc814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:53.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.792+0000 7fabcd9af640 1 -- 172.21.15.64:0/3251030650 shutdown_connections 2024-03-25T18:12:53.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.792+0000 7fabcd9af640 1 -- 172.21.15.64:0/3251030650 wait complete. 2024-03-25T18:12:53.793 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:12:53.792+0000 7fabcd9af640 1 librados: shutdown 2024-03-25T18:12:53.804 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:00.610 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:00.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.721+0000 7f02d6fcd640 10 monclient: get_monmap_and_config 2024-03-25T18:13:00.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.721+0000 7f02d6fcd640 10 monclient: build_initial_monmap 2024-03-25T18:13:00.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.722+0000 7f02d6fcd640 10 monclient: monmap: 2024-03-25T18:13:00.723 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:13:00.723 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:00.723 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:13:00.722860+0000 2024-03-25T18:13:00.723 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:13:00.722860+0000 2024-03-25T18:13:00.723 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:00.723 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:00.723 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:13:00.723 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:13:00.723 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:13:00.723 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:00.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.723+0000 7f02d6fcd640 1 Processor -- start 2024-03-25T18:13:00.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.723+0000 7f02d6fcd640 1 -- start start 2024-03-25T18:13:00.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.723+0000 7f02d6fcd640 10 monclient: init 2024-03-25T18:13:00.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.724+0000 7f02d6fcd640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:00.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.724+0000 7f02d6fcd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:00.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.724+0000 7f02d6fcd640 10 monclient(hunting): picked mon.noname-c con 0x7f02d00a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:13:00.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.724+0000 7f02d6fcd640 10 monclient(hunting): picked mon.noname-b con 0x7f02d013f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:13:00.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.724+0000 7f02d6fcd640 10 monclient(hunting): picked mon.noname-a con 0x7f02d01430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:13:00.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.724+0000 7f02d6fcd640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d0142520 con 0x7f02d01430c0 2024-03-25T18:13:00.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.724+0000 7f02d6fcd640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d00b28a0 con 0x7f02d00a98f0 2024-03-25T18:13:00.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.724+0000 7f02d6fcd640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d00b2a30 con 0x7f02d013f710 2024-03-25T18:13:00.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.724+0000 7f02d6fcd640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:00.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.724+0000 7f02d6fcd640 10 monclient(hunting): authenticate will time out at 7166.962682s 2024-03-25T18:13:00.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.734+0000 7f02cffff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f02d013f710 0x7f02d01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:48058/0 (socket says 172.21.15.64:48058) 2024-03-25T18:13:00.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.734+0000 7f02cffff640 1 -- 172.21.15.64:0/3958845588 learned_addr learned my addr 172.21.15.64:0/3958845588 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:13:00.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.753+0000 7f02cf7fe640 1 -- 172.21.15.64:0/3958845588 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2078236279 0 0) 0x7f02d00b2a30 con 0x7f02d013f710 2024-03-25T18:13:00.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.754+0000 7f02cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:00.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.754+0000 7f02cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:00.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.754+0000 7f02cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:00.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.754+0000 7f02cf7fe640 10 monclient(hunting): my global_id is 4379 2024-03-25T18:13:00.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.754+0000 7f02cf7fe640 1 -- 172.21.15.64:0/3958845588 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c0003520 con 0x7f02d013f710 2024-03-25T18:13:00.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.757+0000 7f02cf7fe640 1 -- 172.21.15.64:0/3958845588 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298948209 0 0) 0x7f02d00b28a0 con 0x7f02d00a98f0 2024-03-25T18:13:00.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.757+0000 7f02cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:00.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.757+0000 7f02cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:00.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.757+0000 7f02cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:00.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.757+0000 7f02cf7fe640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:13:00.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.757+0000 7f02cf7fe640 1 -- 172.21.15.64:0/3958845588 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d00b2a30 con 0x7f02d00a98f0 2024-03-25T18:13:00.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.762+0000 7f02cf7fe640 1 -- 172.21.15.64:0/3958845588 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007988752 0 0) 0x7f02d0142520 con 0x7f02d01430c0 2024-03-25T18:13:00.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.762+0000 7f02cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:00.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.762+0000 7f02cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:00.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.762+0000 7f02cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:00.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.762+0000 7f02cf7fe640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:13:00.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.762+0000 7f02cf7fe640 1 -- 172.21.15.64:0/3958845588 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d00b28a0 con 0x7f02d01430c0 2024-03-25T18:13:00.772 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.771+0000 7f02cf7fe640 1 -- 172.21.15.64:0/3958845588 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1064226425 0 0) 0x7f02d00b2a30 con 0x7f02d00a98f0 2024-03-25T18:13:00.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.771+0000 7f02cf7fe640 1 -- 172.21.15.64:0/3958845588 >> v1:172.21.15.64:6790/0 conn(0x7f02d013f710 legacy=0x7f02d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:00.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.771+0000 7f02cf7fe640 1 -- 172.21.15.64:0/3958845588 >> v1:172.21.15.64:6789/0 conn(0x7f02d01430c0 legacy=0x7f02d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:00.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.771+0000 7f02cf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:00.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.771+0000 7f02cf7fe640 1 monclient: found mon.noname-c 2024-03-25T18:13:00.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.771+0000 7f02cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:00.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.771+0000 7f02cf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:13:00.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.771+0000 7f02cf7fe640 1 -- 172.21.15.64:0/3958845588 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d00b44c0 con 0x7f02d00a98f0 2024-03-25T18:13:00.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.771+0000 7f02cf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:00.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.771+0000 7f02cf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:00.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.771+0000 7f02cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:00.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.771+0000 7f02cf7fe640 1 -- 172.21.15.64:0/3958845588 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f02c4001670 con 0x7f02d00a98f0 2024-03-25T18:13:00.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.772+0000 7f02cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:00.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.772+0000 7f02cf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:00.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.772+0000 7f02cf7fe640 10 monclient: dump: 2024-03-25T18:13:00.773 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:00.774 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:00.774 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:00.774 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:00.774 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:00.774 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:00.774 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:00.774 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:00.774 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:00.774 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:00.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.772+0000 7f02d6fcd640 5 monclient: authenticate success, global_id 4369 2024-03-25T18:13:00.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.772+0000 7f02d6fcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:00.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.783+0000 7f02cf7fe640 1 -- 172.21.15.64:0/3958845588 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02c40017d0 con 0x7f02d00a98f0 2024-03-25T18:13:00.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.783+0000 7f02cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:00.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.783+0000 7f02cf7fe640 1 -- 172.21.15.64:0/3958845588 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f02c4001be0 con 0x7f02d00a98f0 2024-03-25T18:13:00.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.783+0000 7f02d6fcd640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:00.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.783+0000 7f02d6fcd640 4 set_mon_vals no callback set 2024-03-25T18:13:00.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.783+0000 7f02cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:00.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.783+0000 7f02cf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:00.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.783+0000 7f02cf7fe640 10 monclient: dump: 2024-03-25T18:13:00.784 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:00.784 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:00.784 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:00.784 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:00.785 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:00.785 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:00.785 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:00.785 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:00.785 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:00.785 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:00.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.784+0000 7f02d6fcd640 10 monclient: shutdown 2024-03-25T18:13:00.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.784+0000 7f02d6fcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:00.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.784+0000 7f02d6fcd640 1 -- 172.21.15.64:0/3958845588 >> v1:172.21.15.86:6789/0 conn(0x7f02d00a98f0 legacy=0x7f02d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:00.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.784+0000 7f02d6fcd640 1 -- 172.21.15.64:0/3958845588 shutdown_connections 2024-03-25T18:13:00.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.784+0000 7f02d6fcd640 1 -- 172.21.15.64:0/3958845588 >> 172.21.15.64:0/3958845588 conn(0x7f02d00b7ea0 msgr2=0x7f02d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:00.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.784+0000 7f02d6fcd640 1 -- 172.21.15.64:0/3958845588 shutdown_connections 2024-03-25T18:13:00.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.784+0000 7f02d6fcd640 1 -- 172.21.15.64:0/3958845588 wait complete. 2024-03-25T18:13:00.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.785+0000 7f02d6fcd640 10 monclient: build_initial_monmap 2024-03-25T18:13:00.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.785+0000 7f02d6fcd640 10 monclient: monmap: 2024-03-25T18:13:00.787 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:13:00.787 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:00.787 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:13:00.787 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:13:00.787 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:00.787 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:13:00.787 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:13:00.787 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:13:00.787 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:13:00.787 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:00.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.785+0000 7f02d6fcd640 1 librados: starting msgr at 2024-03-25T18:13:00.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.785+0000 7f02d6fcd640 1 librados: starting objecter 2024-03-25T18:13:00.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.785+0000 7f02d6fcd640 1 Processor -- start 2024-03-25T18:13:00.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.785+0000 7f02d6fcd640 1 -- start start 2024-03-25T18:13:00.787 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.785+0000 7f02d6fcd640 1 librados: setting wanted keys 2024-03-25T18:13:00.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.785+0000 7f02d6fcd640 1 librados: calling monclient init 2024-03-25T18:13:00.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.785+0000 7f02d6fcd640 10 monclient: init 2024-03-25T18:13:00.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.786+0000 7f02d6fcd640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:00.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.786+0000 7f02d6fcd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:00.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.786+0000 7f02d6fcd640 10 monclient(hunting): picked mon.noname-c con 0x7f02d00a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:13:00.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.786+0000 7f02d6fcd640 10 monclient(hunting): picked mon.noname-b con 0x7f02d013f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:13:00.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.786+0000 7f02d6fcd640 10 monclient(hunting): picked mon.noname-a con 0x7f02d01430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:13:00.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.786+0000 7f02d6fcd640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d013b180 con 0x7f02d01430c0 2024-03-25T18:13:00.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.786+0000 7f02d6fcd640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d01db890 con 0x7f02d013f710 2024-03-25T18:13:00.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.786+0000 7f02d6fcd640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d00b7b90 con 0x7f02d00a98f0 2024-03-25T18:13:00.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.786+0000 7f02d6fcd640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:00.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.786+0000 7f02d6fcd640 10 monclient(hunting): authenticate will time out at 7167.025124s 2024-03-25T18:13:00.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.794+0000 7f02d4d69640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f02d00a98f0 0x7f02d00b7480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:48096/0 (socket says 172.21.15.64:48096) 2024-03-25T18:13:00.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.794+0000 7f02d4d69640 1 -- 172.21.15.64:0/2043203157 learned_addr learned my addr 172.21.15.64:0/2043203157 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:13:00.798 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.797+0000 7f02cd7fa640 10 client.?.objecter ms_handle_connect 0x7f02d00a98f0 2024-03-25T18:13:00.798 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.797+0000 7f02cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:00.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.803+0000 7f02cd7fa640 1 -- 172.21.15.64:0/2043203157 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885000310 0 0) 0x7f02d00b7b90 con 0x7f02d00a98f0 2024-03-25T18:13:00.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.803+0000 7f02cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:00.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.803+0000 7f02cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:00.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.803+0000 7f02cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:00.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.803+0000 7f02cd7fa640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:13:00.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.803+0000 7f02cd7fa640 1 -- 172.21.15.64:0/2043203157 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02b4004eb0 con 0x7f02d00a98f0 2024-03-25T18:13:00.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.807+0000 7f02cd7fa640 10 client.?.objecter ms_handle_connect 0x7f02d01430c0 2024-03-25T18:13:00.808 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.807+0000 7f02cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:00.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.813+0000 7f02cd7fa640 1 -- 172.21.15.64:0/2043203157 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2919800366 0 0) 0x7f02b4004eb0 con 0x7f02d00a98f0 2024-03-25T18:13:00.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.813+0000 7f02cd7fa640 1 -- 172.21.15.64:0/2043203157 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02d00b7b90 con 0x7f02d00a98f0 2024-03-25T18:13:00.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.814+0000 7f02cd7fa640 1 -- 172.21.15.64:0/2043203157 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f02c4001670 con 0x7f02d00a98f0 2024-03-25T18:13:00.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.814+0000 7f02cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:00.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.814+0000 7f02cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:00.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.814+0000 7f02cd7fa640 10 monclient(hunting): dump: 2024-03-25T18:13:00.815 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:00.815 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:00.815 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:00.815 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:00.815 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:00.815 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:00.815 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:00.815 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:00.815 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:00.815 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:00.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.814+0000 7f02cd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:00.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.815+0000 7f02cd7fa640 10 client.?.objecter ms_handle_connect 0x7f02d013f710 2024-03-25T18:13:00.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.815+0000 7f02cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:00.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.816+0000 7f02cd7fa640 1 -- 172.21.15.64:0/2043203157 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021501789 0 0) 0x7f02d013b180 con 0x7f02d01430c0 2024-03-25T18:13:00.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.816+0000 7f02cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:00.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.816+0000 7f02cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:00.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.816+0000 7f02cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:00.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.816+0000 7f02cd7fa640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:13:00.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.816+0000 7f02cd7fa640 1 -- 172.21.15.64:0/2043203157 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02b4004eb0 con 0x7f02d01430c0 2024-03-25T18:13:00.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.823+0000 7f02cd7fa640 1 -- 172.21.15.64:0/2043203157 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4170803129 0 0) 0x7f02d00b7b90 con 0x7f02d00a98f0 2024-03-25T18:13:00.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.823+0000 7f02cd7fa640 1 -- 172.21.15.64:0/2043203157 >> v1:172.21.15.86:6789/0 conn(0x7f02d013f710 legacy=0x7f02d01daa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:00.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.823+0000 7f02cd7fa640 1 -- 172.21.15.64:0/2043203157 >> v1:172.21.15.64:6789/0 conn(0x7f02d01430c0 legacy=0x7f02d01db180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:00.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.823+0000 7f02cd7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:13:00.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.823+0000 7f02cd7fa640 1 monclient: found mon.c 2024-03-25T18:13:00.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.823+0000 7f02cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:00.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.823+0000 7f02cd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:13:00.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.823+0000 7f02cd7fa640 1 -- 172.21.15.64:0/2043203157 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d013b3f0 con 0x7f02d00a98f0 2024-03-25T18:13:00.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.823+0000 7f02cd7fa640 10 monclient: _finish_auth 0 2024-03-25T18:13:00.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.823+0000 7f02cd7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:13:00.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.823+0000 7f02cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:00.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.823+0000 7f02d6fcd640 5 monclient: authenticate success, global_id 4382 2024-03-25T18:13:00.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.823+0000 7f02d6fcd640 10 monclient: _renew_subs 2024-03-25T18:13:00.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.823+0000 7f02d6fcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:13:00.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.823+0000 7f02d6fcd640 1 -- 172.21.15.64:0/2043203157 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d013a1b0 con 0x7f02d00a98f0 2024-03-25T18:13:00.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.823+0000 7f02d6fcd640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:00.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.824+0000 7f02d6fcd640 10 monclient: _renew_subs 2024-03-25T18:13:00.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.824+0000 7f02d6fcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:13:00.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.824+0000 7f02d6fcd640 1 -- 172.21.15.64:0/2043203157 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d013ba40 con 0x7f02d00a98f0 2024-03-25T18:13:00.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.824+0000 7f02d6fcd640 1 librados: init done 2024-03-25T18:13:00.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.825+0000 7f02d6fcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:00.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.825+0000 7f02d6fcd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:00.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.825+0000 7f02d6fcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:13:00.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:00.825+0000 7f02d6fcd640 1 -- 172.21.15.64:0/2043203157 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d00b3ef0 con 0x7f02d00a98f0 2024-03-25T18:13:01.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.047+0000 7f02cd7fa640 1 -- 172.21.15.64:0/2043203157 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02c40038a0 con 0x7f02d00a98f0 2024-03-25T18:13:01.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.047+0000 7f02cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:01.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.047+0000 7f02cd7fa640 1 -- 172.21.15.64:0/2043203157 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f02c4003c70 con 0x7f02d00a98f0 2024-03-25T18:13:01.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.047+0000 7f02cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:01.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.047+0000 7f02cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:01.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.047+0000 7f02cd7fa640 10 monclient: dump: 2024-03-25T18:13:01.048 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:01.048 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:01.048 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:01.048 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:01.048 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:01.048 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:01.048 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:01.048 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:01.048 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:01.049 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:01.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.047+0000 7f02ceffd640 4 set_mon_vals no callback set 2024-03-25T18:13:01.133 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.133+0000 7f02cd7fa640 1 -- 172.21.15.64:0/2043203157 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f02c4003dd0 con 0x7f02d00a98f0 2024-03-25T18:13:01.134 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.133+0000 7f02cffff640 10 monclient: get_auth_request con 0x7f02b406dd90 auth_method 0 2024-03-25T18:13:01.134 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.133+0000 7f02cd7fa640 10 client.4382.objecter ms_handle_connect 0x7f02b406dd90 2024-03-25T18:13:01.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.140+0000 7f02cd7fa640 1 -- 172.21.15.64:0/2043203157 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f02c4054940 con 0x7f02d00a98f0 2024-03-25T18:13:01.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.140+0000 7f02cd7fa640 10 client.4382.objecter ms_dispatch 0x7f02d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:01.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.140+0000 7f02cd7fa640 3 client.4382.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:01.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.140+0000 7f02cd7fa640 3 client.4382.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:01.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.140+0000 7f02cd7fa640 20 client.4382.objecter dump_active .. 0 homeless 2024-03-25T18:13:01.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.140+0000 7f02cd7fa640 1 -- 172.21.15.64:0/2043203157 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02c4050450 con 0x7f02d00a98f0 2024-03-25T18:13:01.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.140+0000 7f02cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:01.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.140+0000 7f02cd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:01.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.355+0000 7f02d6fcd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:01.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.355+0000 7f02d6fcd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:01.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.355+0000 7f02d6fcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:13:01.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.355+0000 7f02d6fcd640 1 -- 172.21.15.64:0/2043203157 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02d00627b0 con 0x7f02d00a98f0 2024-03-25T18:13:01.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.386+0000 7f02cd7fa640 1 -- 172.21.15.64:0/2043203157 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f02c4052380 con 0x7f02d00a98f0 2024-03-25T18:13:01.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.386+0000 7f02cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:01.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.386+0000 7f02cd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:01.388 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:13:01.388 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:01.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.388+0000 7f02d6fcd640 10 librados: watch_flush enter 2024-03-25T18:13:01.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.388+0000 7f02d6fcd640 10 librados: watch_flush exit 2024-03-25T18:13:01.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.388+0000 7f02d6fcd640 20 client.4382.objecter shutdown clearing up homeless session... 2024-03-25T18:13:01.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.388+0000 7f02d6fcd640 10 client.4382.objecter successfully canceled tick 2024-03-25T18:13:01.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.389+0000 7f02d6fcd640 1 -- 172.21.15.64:0/2043203157 >> v1:172.21.15.64:6800/2915543638 conn(0x7f02b406dd90 legacy=0x7f02b4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:01.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.389+0000 7f02d6fcd640 10 monclient: shutdown 2024-03-25T18:13:01.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.389+0000 7f02d6fcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:01.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.389+0000 7f02d6fcd640 1 -- 172.21.15.64:0/2043203157 >> v1:172.21.15.64:6790/0 conn(0x7f02d00a98f0 legacy=0x7f02d00b7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:01.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.389+0000 7f02d6fcd640 1 -- 172.21.15.64:0/2043203157 shutdown_connections 2024-03-25T18:13:01.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.389+0000 7f02d6fcd640 1 -- 172.21.15.64:0/2043203157 >> 172.21.15.64:0/2043203157 conn(0x7f02d00b7ea0 msgr2=0x7f02d0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:01.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.389+0000 7f02d6fcd640 1 -- 172.21.15.64:0/2043203157 shutdown_connections 2024-03-25T18:13:01.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.389+0000 7f02d6fcd640 1 -- 172.21.15.64:0/2043203157 wait complete. 2024-03-25T18:13:01.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:01.389+0000 7f02d6fcd640 1 librados: shutdown 2024-03-25T18:13:01.401 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:08.207 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:08.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.319+0000 7f5e0d532640 10 monclient: get_monmap_and_config 2024-03-25T18:13:08.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.319+0000 7f5e0d532640 10 monclient: build_initial_monmap 2024-03-25T18:13:08.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.319+0000 7f5e0d532640 10 monclient: monmap: 2024-03-25T18:13:08.320 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:13:08.320 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:08.320 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:13:08.320433+0000 2024-03-25T18:13:08.320 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:13:08.320433+0000 2024-03-25T18:13:08.320 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:08.320 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:08.320 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:13:08.321 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:13:08.321 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:13:08.321 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:08.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.320+0000 7f5e0d532640 1 Processor -- start 2024-03-25T18:13:08.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.320+0000 7f5e0d532640 1 -- start start 2024-03-25T18:13:08.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.320+0000 7f5e0d532640 10 monclient: init 2024-03-25T18:13:08.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.320+0000 7f5e0d532640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:08.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.320+0000 7f5e0d532640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:08.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.320+0000 7f5e0d532640 10 monclient(hunting): picked mon.noname-a con 0x7f5e0806adf0 addr v1:172.21.15.64:6789/0 2024-03-25T18:13:08.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.321+0000 7f5e0d532640 10 monclient(hunting): picked mon.noname-c con 0x7f5e080aee20 addr v1:172.21.15.86:6789/0 2024-03-25T18:13:08.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.321+0000 7f5e0d532640 10 monclient(hunting): picked mon.noname-b con 0x7f5e080afe80 addr v1:172.21.15.64:6790/0 2024-03-25T18:13:08.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.321+0000 7f5e0d532640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e080729c0 con 0x7f5e0806adf0 2024-03-25T18:13:08.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.321+0000 7f5e0d532640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e08143e60 con 0x7f5e080aee20 2024-03-25T18:13:08.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.321+0000 7f5e0d532640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0813db40 con 0x7f5e080afe80 2024-03-25T18:13:08.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.321+0000 7f5e0d532640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:08.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.321+0000 7f5e0d532640 10 monclient(hunting): authenticate will time out at 7174.559565s 2024-03-25T18:13:08.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.326+0000 7f5e077fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f5e080afe80 0x7f5e0814baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:47586/0 (socket says 172.21.15.64:47586) 2024-03-25T18:13:08.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.326+0000 7f5e077fe640 1 -- 172.21.15.64:0/1418465217 learned_addr learned my addr 172.21.15.64:0/1418465217 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:13:08.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.337+0000 7f5e05ffb640 1 -- 172.21.15.64:0/1418465217 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472635783 0 0) 0x7f5e0813db40 con 0x7f5e080afe80 2024-03-25T18:13:08.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.337+0000 7f5e05ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:08.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.337+0000 7f5e05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:08.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.337+0000 7f5e05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:08.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.337+0000 7f5e05ffb640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:13:08.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.337+0000 7f5e05ffb640 1 -- 172.21.15.64:0/1418465217 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df0003520 con 0x7f5e080afe80 2024-03-25T18:13:08.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.350+0000 7f5e05ffb640 1 -- 172.21.15.64:0/1418465217 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098002774 0 0) 0x7f5e08143e60 con 0x7f5e080aee20 2024-03-25T18:13:08.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.350+0000 7f5e05ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:08.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.350+0000 7f5e05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:08.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.350+0000 7f5e05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:08.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.350+0000 7f5e05ffb640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:13:08.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.350+0000 7f5e05ffb640 1 -- 172.21.15.64:0/1418465217 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e0813db40 con 0x7f5e080aee20 2024-03-25T18:13:08.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.354+0000 7f5e05ffb640 1 -- 172.21.15.64:0/1418465217 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1375798694 0 0) 0x7f5df0003520 con 0x7f5e080afe80 2024-03-25T18:13:08.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.354+0000 7f5e05ffb640 1 -- 172.21.15.64:0/1418465217 >> v1:172.21.15.86:6789/0 conn(0x7f5e080aee20 legacy=0x7f5e0813d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:08.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.354+0000 7f5e05ffb640 1 -- 172.21.15.64:0/1418465217 >> v1:172.21.15.64:6789/0 conn(0x7f5e0806adf0 legacy=0x7f5e08143750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:08.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.354+0000 7f5e05ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:08.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.354+0000 7f5e05ffb640 1 monclient: found mon.noname-b 2024-03-25T18:13:08.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.354+0000 7f5e05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:08.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.354+0000 7f5e05ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:13:08.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.354+0000 7f5e05ffb640 1 -- 172.21.15.64:0/1418465217 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0813dcd0 con 0x7f5e080afe80 2024-03-25T18:13:08.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.354+0000 7f5e05ffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:08.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.354+0000 7f5e05ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:08.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.354+0000 7f5e05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:08.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.354+0000 7f5e05ffb640 1 -- 172.21.15.64:0/1418465217 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f5e00001a10 con 0x7f5e080afe80 2024-03-25T18:13:08.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.354+0000 7f5e0d532640 5 monclient: authenticate success, global_id 4385 2024-03-25T18:13:08.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.354+0000 7f5e0d532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:08.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.354+0000 7f5e05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:08.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.354+0000 7f5e05ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:08.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.354+0000 7f5e05ffb640 10 monclient: dump: 2024-03-25T18:13:08.356 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:08.356 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:08.356 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:08.356 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:08.356 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:08.356 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:08.356 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:08.356 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:08.357 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:08.357 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:08.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.354+0000 7f5e0d532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:08.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.360+0000 7f5e05ffb640 1 -- 172.21.15.64:0/1418465217 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e00001b70 con 0x7f5e080afe80 2024-03-25T18:13:08.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.360+0000 7f5e05ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:08.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.360+0000 7f5e05ffb640 1 -- 172.21.15.64:0/1418465217 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f5e00001f80 con 0x7f5e080afe80 2024-03-25T18:13:08.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.360+0000 7f5e0d532640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:08.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.360+0000 7f5e0d532640 4 set_mon_vals no callback set 2024-03-25T18:13:08.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.360+0000 7f5e05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:08.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.360+0000 7f5e05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:08.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.360+0000 7f5e05ffb640 10 monclient: dump: 2024-03-25T18:13:08.361 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:08.361 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:08.361 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:08.361 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:08.361 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:08.361 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:08.361 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:08.361 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:08.361 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:08.362 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:08.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.360+0000 7f5e0d532640 10 monclient: shutdown 2024-03-25T18:13:08.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.360+0000 7f5e0d532640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:08.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.360+0000 7f5e0d532640 1 -- 172.21.15.64:0/1418465217 >> v1:172.21.15.64:6790/0 conn(0x7f5e080afe80 legacy=0x7f5e0814baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:08.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.360+0000 7f5e0d532640 1 -- 172.21.15.64:0/1418465217 shutdown_connections 2024-03-25T18:13:08.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.360+0000 7f5e0d532640 1 -- 172.21.15.64:0/1418465217 >> 172.21.15.64:0/1418465217 conn(0x7f5e080b4a80 msgr2=0x7f5e080b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:08.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.361+0000 7f5e0d532640 1 -- 172.21.15.64:0/1418465217 shutdown_connections 2024-03-25T18:13:08.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.361+0000 7f5e0d532640 1 -- 172.21.15.64:0/1418465217 wait complete. 2024-03-25T18:13:08.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.361+0000 7f5e0d532640 10 monclient: build_initial_monmap 2024-03-25T18:13:08.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.361+0000 7f5e0d532640 10 monclient: monmap: 2024-03-25T18:13:08.364 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:13:08.364 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:08.364 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:13:08.364 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:13:08.365 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:08.365 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:13:08.365 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:13:08.365 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:13:08.365 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:13:08.365 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:08.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.361+0000 7f5e0d532640 1 librados: starting msgr at 2024-03-25T18:13:08.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.361+0000 7f5e0d532640 1 librados: starting objecter 2024-03-25T18:13:08.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.362+0000 7f5e0d532640 1 Processor -- start 2024-03-25T18:13:08.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.362+0000 7f5e0d532640 1 -- start start 2024-03-25T18:13:08.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.362+0000 7f5e0d532640 1 librados: setting wanted keys 2024-03-25T18:13:08.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.362+0000 7f5e0d532640 1 librados: calling monclient init 2024-03-25T18:13:08.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.362+0000 7f5e0d532640 10 monclient: init 2024-03-25T18:13:08.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.362+0000 7f5e0d532640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:08.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.362+0000 7f5e0d532640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:08.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.362+0000 7f5e0d532640 10 monclient(hunting): picked mon.noname-a con 0x7f5e0806adf0 addr v1:172.21.15.64:6789/0 2024-03-25T18:13:08.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.362+0000 7f5e0d532640 10 monclient(hunting): picked mon.noname-c con 0x7f5e080afe80 addr v1:172.21.15.64:6790/0 2024-03-25T18:13:08.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.362+0000 7f5e0d532640 10 monclient(hunting): picked mon.noname-b con 0x7f5e080ac570 addr v1:172.21.15.86:6789/0 2024-03-25T18:13:08.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.363+0000 7f5e0d532640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e080ad120 con 0x7f5e0806adf0 2024-03-25T18:13:08.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.363+0000 7f5e0d532640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e080ade80 con 0x7f5e080ac570 2024-03-25T18:13:08.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.363+0000 7f5e0d532640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e080abb00 con 0x7f5e080afe80 2024-03-25T18:13:08.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.363+0000 7f5e0d532640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:08.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.363+0000 7f5e0d532640 10 monclient(hunting): authenticate will time out at 7174.601673s 2024-03-25T18:13:08.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.369+0000 7f5e077fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f5e080ac570 0x7f5e081e0af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:48770/0 (socket says 172.21.15.64:48770) 2024-03-25T18:13:08.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.369+0000 7f5e077fe640 1 -- 172.21.15.64:0/2103788568 learned_addr learned my addr 172.21.15.64:0/2103788568 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:13:08.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.374+0000 7f5def7fe640 10 client.?.objecter ms_handle_connect 0x7f5e080ac570 2024-03-25T18:13:08.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.374+0000 7f5def7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:08.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.382+0000 7f5def7fe640 10 client.?.objecter ms_handle_connect 0x7f5e080afe80 2024-03-25T18:13:08.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.382+0000 7f5def7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:08.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.388+0000 7f5def7fe640 10 client.?.objecter ms_handle_connect 0x7f5e0806adf0 2024-03-25T18:13:08.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.388+0000 7f5def7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:08.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.389+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996013610 0 0) 0x7f5e080ade80 con 0x7f5e080ac570 2024-03-25T18:13:08.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.389+0000 7f5def7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:08.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.389+0000 7f5def7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:08.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.389+0000 7f5def7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:08.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.389+0000 7f5def7fe640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:13:08.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.389+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df4004eb0 con 0x7f5e080ac570 2024-03-25T18:13:08.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.399+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174434082 0 0) 0x7f5e080ad120 con 0x7f5e0806adf0 2024-03-25T18:13:08.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.399+0000 7f5def7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:08.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.399+0000 7f5def7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:08.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.399+0000 7f5def7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:08.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.399+0000 7f5def7fe640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:13:08.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.399+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e080ade80 con 0x7f5e0806adf0 2024-03-25T18:13:08.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.403+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094144956 0 0) 0x7f5e080abb00 con 0x7f5e080afe80 2024-03-25T18:13:08.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.403+0000 7f5def7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:08.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.403+0000 7f5def7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:08.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.403+0000 7f5def7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:08.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.403+0000 7f5def7fe640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:13:08.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.403+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e080ad120 con 0x7f5e080afe80 2024-03-25T18:13:08.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.412+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 859645867 0 0) 0x7f5df4004eb0 con 0x7f5e080ac570 2024-03-25T18:13:08.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.413+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e080abb00 con 0x7f5e080ac570 2024-03-25T18:13:08.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.413+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f5e00001ec0 con 0x7f5e080ac570 2024-03-25T18:13:08.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.413+0000 7f5def7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:08.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.413+0000 7f5def7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:08.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.413+0000 7f5def7fe640 10 monclient(hunting): dump: 2024-03-25T18:13:08.414 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:08.414 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:08.414 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:08.414 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:08.414 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:08.414 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:08.414 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:08.414 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:08.414 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:08.414 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:08.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.413+0000 7f5def7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:08.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.423+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3123277464 0 0) 0x7f5e080ad120 con 0x7f5e080afe80 2024-03-25T18:13:08.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.423+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5df4004eb0 con 0x7f5e080afe80 2024-03-25T18:13:08.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.423+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f5dfc001930 con 0x7f5e080afe80 2024-03-25T18:13:08.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.423+0000 7f5def7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:08.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.423+0000 7f5def7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:08.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.423+0000 7f5def7fe640 10 monclient(hunting): dump: 2024-03-25T18:13:08.424 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:08.425 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:08.425 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:08.425 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:08.425 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:08.425 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:08.425 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:08.425 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:08.425 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:08.425 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:08.429 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.429+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 77826168 0 0) 0x7f5e080ade80 con 0x7f5e0806adf0 2024-03-25T18:13:08.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.429+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e080ad120 con 0x7f5e0806adf0 2024-03-25T18:13:08.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.429+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f5df80018b0 con 0x7f5e0806adf0 2024-03-25T18:13:08.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.429+0000 7f5def7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:08.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.429+0000 7f5def7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:08.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.429+0000 7f5def7fe640 10 monclient(hunting): dump: 2024-03-25T18:13:08.430 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:08.430 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:08.430 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:08.430 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:08.430 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:08.430 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:08.430 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:08.430 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:08.430 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:08.431 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:08.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 316559967 0 0) 0x7f5df4004eb0 con 0x7f5e080afe80 2024-03-25T18:13:08.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 >> v1:172.21.15.86:6789/0 conn(0x7f5e080ac570 legacy=0x7f5e081e0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:08.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 >> v1:172.21.15.64:6789/0 conn(0x7f5e0806adf0 legacy=0x7f5e08138fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:08.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5def7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:08.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5def7fe640 1 monclient: found mon.c 2024-03-25T18:13:08.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5def7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:08.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5def7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:13:08.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e080acc00 con 0x7f5e080afe80 2024-03-25T18:13:08.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5def7fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:08.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5def7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:08.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5def7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:08.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5e0d532640 5 monclient: authenticate success, global_id 4388 2024-03-25T18:13:08.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5e0d532640 10 monclient: _renew_subs 2024-03-25T18:13:08.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5e0d532640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:13:08.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5e0d532640 1 -- 172.21.15.64:0/2103788568 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e081396b0 con 0x7f5e080afe80 2024-03-25T18:13:08.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5e0d532640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:08.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5e0d532640 10 monclient: _renew_subs 2024-03-25T18:13:08.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5e0d532640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:13:08.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5e0d532640 1 -- 172.21.15.64:0/2103788568 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e080ae7f0 con 0x7f5e080afe80 2024-03-25T18:13:08.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.433+0000 7f5e0d532640 1 librados: init done 2024-03-25T18:13:08.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.434+0000 7f5e0d532640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:08.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.434+0000 7f5e0d532640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:08.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.434+0000 7f5e0d532640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:13:08.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.434+0000 7f5e0d532640 1 -- 172.21.15.64:0/2103788568 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e080aca60 con 0x7f5e080afe80 2024-03-25T18:13:08.439 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.439+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5dfc003610 con 0x7f5e080afe80 2024-03-25T18:13:08.439 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.439+0000 7f5def7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:08.439 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.439+0000 7f5e057fa640 4 set_mon_vals no callback set 2024-03-25T18:13:08.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.750+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f5dfc001d80 con 0x7f5e080afe80 2024-03-25T18:13:08.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.750+0000 7f5def7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:08.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.750+0000 7f5def7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:08.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.750+0000 7f5def7fe640 10 monclient: dump: 2024-03-25T18:13:08.751 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:08.751 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:08.751 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:08.751 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:08.751 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:08.751 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:08.751 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:08.751 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:08.752 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:08.752 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:08.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.751+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f5dfc019670 con 0x7f5e080afe80 2024-03-25T18:13:08.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.751+0000 7f5e06ffd640 10 monclient: get_auth_request con 0x7f5df406e160 auth_method 0 2024-03-25T18:13:08.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.751+0000 7f5def7fe640 10 client.4388.objecter ms_handle_connect 0x7f5df406e160 2024-03-25T18:13:08.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.757+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f5dfc0533e0 con 0x7f5e080afe80 2024-03-25T18:13:08.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.757+0000 7f5def7fe640 10 client.4388.objecter ms_dispatch 0x7f5e08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:08.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.757+0000 7f5def7fe640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:08.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.757+0000 7f5def7fe640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:08.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.757+0000 7f5def7fe640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-25T18:13:08.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.757+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5dfc04ed20 con 0x7f5e080afe80 2024-03-25T18:13:08.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.757+0000 7f5def7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:08.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:08.757+0000 7f5def7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:09.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.013+0000 7f5e0d532640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:09.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.013+0000 7f5e0d532640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:09.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.013+0000 7f5e0d532640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:13:09.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.013+0000 7f5e0d532640 1 -- 172.21.15.64:0/2103788568 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e080acdb0 con 0x7f5e080afe80 2024-03-25T18:13:09.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.046+0000 7f5def7fe640 1 -- 172.21.15.64:0/2103788568 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f5e080acdb0 con 0x7f5e080afe80 2024-03-25T18:13:09.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.046+0000 7f5def7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:09.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.046+0000 7f5def7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:09.047 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:13:09.047 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:09.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.048+0000 7f5e0d532640 10 librados: watch_flush enter 2024-03-25T18:13:09.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.048+0000 7f5e0d532640 10 librados: watch_flush exit 2024-03-25T18:13:09.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.048+0000 7f5e0d532640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-25T18:13:09.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.048+0000 7f5e0d532640 10 client.4388.objecter successfully canceled tick 2024-03-25T18:13:09.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.048+0000 7f5e0d532640 1 -- 172.21.15.64:0/2103788568 >> v1:172.21.15.64:6800/2915543638 conn(0x7f5df406e160 legacy=0x7f5df4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:09.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.049+0000 7f5e0d532640 10 monclient: shutdown 2024-03-25T18:13:09.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.049+0000 7f5e0d532640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:09.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.049+0000 7f5e0d532640 1 -- 172.21.15.64:0/2103788568 >> v1:172.21.15.64:6790/0 conn(0x7f5e080afe80 legacy=0x7f5e081d87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:09.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.049+0000 7f5e0d532640 1 -- 172.21.15.64:0/2103788568 shutdown_connections 2024-03-25T18:13:09.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.049+0000 7f5e0d532640 1 -- 172.21.15.64:0/2103788568 >> 172.21.15.64:0/2103788568 conn(0x7f5e080b4a80 msgr2=0x7f5e08143180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:09.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.049+0000 7f5e0d532640 1 -- 172.21.15.64:0/2103788568 shutdown_connections 2024-03-25T18:13:09.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.049+0000 7f5e0d532640 1 -- 172.21.15.64:0/2103788568 wait complete. 2024-03-25T18:13:09.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:09.049+0000 7f5e0d532640 1 librados: shutdown 2024-03-25T18:13:09.062 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:11.898 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:03:55.011 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:11.902 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:03:54.981 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:11.919 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:03:55.032 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:11.958 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:03:55.070 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:11.958 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:03:55.037 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:11.970 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:03:55.082 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:11.975 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:03:55.088 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:11.988 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:03:55.067 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:12.013 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:03:55.126 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:15.867 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:15.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:15.983+0000 7fcf69313640 10 monclient: get_monmap_and_config 2024-03-25T18:13:15.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:15.983+0000 7fcf69313640 10 monclient: build_initial_monmap 2024-03-25T18:13:15.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:15.983+0000 7fcf69313640 10 monclient: monmap: 2024-03-25T18:13:15.984 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:13:15.984 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:15.984 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:13:15.984280+0000 2024-03-25T18:13:15.984 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:13:15.984280+0000 2024-03-25T18:13:15.984 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:15.984 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:15.985 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:13:15.985 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:13:15.985 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:13:15.985 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:15.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:15.983+0000 7fcf69313640 1 Processor -- start 2024-03-25T18:13:15.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:15.984+0000 7fcf69313640 1 -- start start 2024-03-25T18:13:15.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:15.984+0000 7fcf69313640 10 monclient: init 2024-03-25T18:13:15.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:15.984+0000 7fcf69313640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:15.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:15.984+0000 7fcf69313640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:15.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:15.984+0000 7fcf69313640 10 monclient(hunting): picked mon.noname-b con 0x7fcf640a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:13:15.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:15.984+0000 7fcf69313640 10 monclient(hunting): picked mon.noname-c con 0x7fcf6413f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:13:15.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:15.984+0000 7fcf69313640 10 monclient(hunting): picked mon.noname-a con 0x7fcf641430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:13:15.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:15.984+0000 7fcf69313640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf64142520 con 0x7fcf641430c0 2024-03-25T18:13:15.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:15.984+0000 7fcf69313640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf640b28a0 con 0x7fcf6413f710 2024-03-25T18:13:15.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:15.984+0000 7fcf69313640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf640b2a30 con 0x7fcf640a98f0 2024-03-25T18:13:15.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:15.984+0000 7fcf69313640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:15.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:15.984+0000 7fcf69313640 10 monclient(hunting): authenticate will time out at 7182.223143s 2024-03-25T18:13:16.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.001+0000 7fcf637fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fcf641430c0 0x7fcf6414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:58624/0 (socket says 172.21.15.64:58624) 2024-03-25T18:13:16.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.001+0000 7fcf637fe640 1 -- 172.21.15.64:0/494691303 learned_addr learned my addr 172.21.15.64:0/494691303 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:13:16.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.008+0000 7fcf61ffb640 1 -- 172.21.15.64:0/494691303 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280167471 0 0) 0x7fcf64142520 con 0x7fcf641430c0 2024-03-25T18:13:16.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.008+0000 7fcf61ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:16.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.008+0000 7fcf61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:16.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.008+0000 7fcf61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:16.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.008+0000 7fcf61ffb640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:13:16.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.008+0000 7fcf61ffb640 1 -- 172.21.15.64:0/494691303 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf4c003520 con 0x7fcf641430c0 2024-03-25T18:13:16.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.015+0000 7fcf61ffb640 1 -- 172.21.15.64:0/494691303 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1383066215 0 0) 0x7fcf4c003520 con 0x7fcf641430c0 2024-03-25T18:13:16.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.015+0000 7fcf61ffb640 1 -- 172.21.15.64:0/494691303 >> v1:172.21.15.64:6790/0 conn(0x7fcf640a98f0 legacy=0x7fcf640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:16.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.015+0000 7fcf61ffb640 1 -- 172.21.15.64:0/494691303 >> v1:172.21.15.86:6789/0 conn(0x7fcf6413f710 legacy=0x7fcf641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:16.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.015+0000 7fcf61ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:16.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.015+0000 7fcf61ffb640 1 monclient: found mon.noname-a 2024-03-25T18:13:16.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.015+0000 7fcf61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:16.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.015+0000 7fcf61ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:13:16.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.015+0000 7fcf61ffb640 1 -- 172.21.15.64:0/494691303 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf640b44c0 con 0x7fcf641430c0 2024-03-25T18:13:16.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.015+0000 7fcf61ffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:16.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.015+0000 7fcf61ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:16.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.015+0000 7fcf61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:16.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.015+0000 7fcf61ffb640 1 -- 172.21.15.64:0/494691303 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcf5c001a10 con 0x7fcf641430c0 2024-03-25T18:13:16.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.015+0000 7fcf69313640 5 monclient: authenticate success, global_id 4386 2024-03-25T18:13:16.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.015+0000 7fcf61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:16.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.015+0000 7fcf61ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:16.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.015+0000 7fcf61ffb640 10 monclient: dump: 2024-03-25T18:13:16.017 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:16.017 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:16.017 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:16.017 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:16.017 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:16.017 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:16.017 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:16.017 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:16.017 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:16.017 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:16.017 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.015+0000 7fcf69313640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:16.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.027+0000 7fcf61ffb640 1 -- 172.21.15.64:0/494691303 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf5c001b70 con 0x7fcf641430c0 2024-03-25T18:13:16.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.027+0000 7fcf61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:16.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.027+0000 7fcf61ffb640 1 -- 172.21.15.64:0/494691303 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcf5c001f80 con 0x7fcf641430c0 2024-03-25T18:13:16.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.027+0000 7fcf69313640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:16.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.027+0000 7fcf69313640 4 set_mon_vals no callback set 2024-03-25T18:13:16.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.027+0000 7fcf61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:16.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.027+0000 7fcf61ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:16.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.027+0000 7fcf61ffb640 10 monclient: dump: 2024-03-25T18:13:16.028 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:16.028 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:16.028 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:16.028 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:16.028 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:16.029 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:16.029 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:16.029 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:16.029 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:16.029 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:16.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.028+0000 7fcf69313640 10 monclient: shutdown 2024-03-25T18:13:16.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.028+0000 7fcf69313640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:16.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.028+0000 7fcf69313640 1 -- 172.21.15.64:0/494691303 >> v1:172.21.15.64:6789/0 conn(0x7fcf641430c0 legacy=0x7fcf6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:16.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.028+0000 7fcf69313640 1 -- 172.21.15.64:0/494691303 shutdown_connections 2024-03-25T18:13:16.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.028+0000 7fcf69313640 1 -- 172.21.15.64:0/494691303 >> 172.21.15.64:0/494691303 conn(0x7fcf640b7ea0 msgr2=0x7fcf640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:16.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.028+0000 7fcf69313640 1 -- 172.21.15.64:0/494691303 shutdown_connections 2024-03-25T18:13:16.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.028+0000 7fcf69313640 1 -- 172.21.15.64:0/494691303 wait complete. 2024-03-25T18:13:16.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.029+0000 7fcf69313640 10 monclient: build_initial_monmap 2024-03-25T18:13:16.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.029+0000 7fcf69313640 10 monclient: monmap: 2024-03-25T18:13:16.031 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:13:16.031 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:16.031 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:13:16.031 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:13:16.031 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:16.031 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:13:16.031 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:13:16.031 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:13:16.031 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:13:16.031 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:16.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.029+0000 7fcf69313640 1 librados: starting msgr at 2024-03-25T18:13:16.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.029+0000 7fcf69313640 1 librados: starting objecter 2024-03-25T18:13:16.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.029+0000 7fcf69313640 1 Processor -- start 2024-03-25T18:13:16.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.029+0000 7fcf69313640 1 -- start start 2024-03-25T18:13:16.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.029+0000 7fcf69313640 1 librados: setting wanted keys 2024-03-25T18:13:16.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.029+0000 7fcf69313640 1 librados: calling monclient init 2024-03-25T18:13:16.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.029+0000 7fcf69313640 10 monclient: init 2024-03-25T18:13:16.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.030+0000 7fcf69313640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:16.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.030+0000 7fcf69313640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:16.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.030+0000 7fcf69313640 10 monclient(hunting): picked mon.noname-a con 0x7fcf640a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:13:16.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.030+0000 7fcf69313640 10 monclient(hunting): picked mon.noname-b con 0x7fcf6413f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:13:16.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.030+0000 7fcf69313640 10 monclient(hunting): picked mon.noname-c con 0x7fcf641430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:13:16.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.030+0000 7fcf69313640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf64138b60 con 0x7fcf640a98f0 2024-03-25T18:13:16.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.030+0000 7fcf69313640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf64137e40 con 0x7fcf6413f710 2024-03-25T18:13:16.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.030+0000 7fcf69313640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf641d82f0 con 0x7fcf641430c0 2024-03-25T18:13:16.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.030+0000 7fcf69313640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:16.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.030+0000 7fcf69313640 10 monclient(hunting): authenticate will time out at 7182.269347s 2024-03-25T18:13:16.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.043+0000 7fcf627fc640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fcf6413f710 0x7fcf641d7be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:53420/0 (socket says 172.21.15.64:53420) 2024-03-25T18:13:16.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.043+0000 7fcf627fc640 1 -- 172.21.15.64:0/1888035775 learned_addr learned my addr 172.21.15.64:0/1888035775 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:13:16.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.044+0000 7fcf4b7fe640 10 client.?.objecter ms_handle_connect 0x7fcf6413f710 2024-03-25T18:13:16.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.044+0000 7fcf4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:16.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.051+0000 7fcf4b7fe640 10 client.?.objecter ms_handle_connect 0x7fcf640a98f0 2024-03-25T18:13:16.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.051+0000 7fcf4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:16.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.057+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262866449 0 0) 0x7fcf64137e40 con 0x7fcf6413f710 2024-03-25T18:13:16.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.057+0000 7fcf4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:16.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.057+0000 7fcf4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:16.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.057+0000 7fcf4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:16.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.057+0000 7fcf4b7fe640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:13:16.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.057+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf50004eb0 con 0x7fcf6413f710 2024-03-25T18:13:16.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.062+0000 7fcf4b7fe640 10 client.?.objecter ms_handle_connect 0x7fcf641430c0 2024-03-25T18:13:16.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.062+0000 7fcf4b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:16.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.076+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820309817 0 0) 0x7fcf64138b60 con 0x7fcf640a98f0 2024-03-25T18:13:16.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.076+0000 7fcf4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:16.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.076+0000 7fcf4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:16.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.076+0000 7fcf4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:16.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.076+0000 7fcf4b7fe640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:13:16.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.076+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf64137e40 con 0x7fcf640a98f0 2024-03-25T18:13:16.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.083+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916762296 0 0) 0x7fcf641d82f0 con 0x7fcf641430c0 2024-03-25T18:13:16.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.083+0000 7fcf4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:16.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.083+0000 7fcf4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:16.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.083+0000 7fcf4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:16.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.083+0000 7fcf4b7fe640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:13:16.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.083+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf64138b60 con 0x7fcf641430c0 2024-03-25T18:13:16.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.090+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1078023967 0 0) 0x7fcf50004eb0 con 0x7fcf6413f710 2024-03-25T18:13:16.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.090+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf641d82f0 con 0x7fcf6413f710 2024-03-25T18:13:16.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.090+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcf580019d0 con 0x7fcf6413f710 2024-03-25T18:13:16.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.090+0000 7fcf4b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:16.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.090+0000 7fcf4b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:16.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.090+0000 7fcf4b7fe640 10 monclient(hunting): dump: 2024-03-25T18:13:16.091 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:16.091 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:16.091 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:16.091 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:16.091 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:16.091 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:16.091 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:16.091 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:16.091 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:16.091 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:16.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.090+0000 7fcf4b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:16.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3298920090 0 0) 0x7fcf641d82f0 con 0x7fcf6413f710 2024-03-25T18:13:16.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 >> v1:172.21.15.64:6790/0 conn(0x7fcf641430c0 legacy=0x7fcf64133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:16.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 >> v1:172.21.15.64:6789/0 conn(0x7fcf640a98f0 legacy=0x7fcf64137730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:16.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf4b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:16.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf4b7fe640 1 monclient: found mon.b 2024-03-25T18:13:16.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:16.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf4b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:13:16.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf64134e70 con 0x7fcf6413f710 2024-03-25T18:13:16.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf4b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:16.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf4b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:16.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:16.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf69313640 5 monclient: authenticate success, global_id 4384 2024-03-25T18:13:16.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf69313640 10 monclient: _renew_subs 2024-03-25T18:13:16.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf69313640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:13:16.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf69313640 1 -- 172.21.15.64:0/1888035775 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf641350c0 con 0x7fcf6413f710 2024-03-25T18:13:16.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf69313640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:16.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf69313640 10 monclient: _renew_subs 2024-03-25T18:13:16.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf69313640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:13:16.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf69313640 1 -- 172.21.15.64:0/1888035775 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf64134800 con 0x7fcf6413f710 2024-03-25T18:13:16.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.101+0000 7fcf69313640 1 librados: init done 2024-03-25T18:13:16.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.102+0000 7fcf69313640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:16.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.102+0000 7fcf69313640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:16.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.102+0000 7fcf69313640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:13:16.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.102+0000 7fcf69313640 1 -- 172.21.15.64:0/1888035775 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf6414f330 con 0x7fcf6413f710 2024-03-25T18:13:16.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.377+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf580035f0 con 0x7fcf6413f710 2024-03-25T18:13:16.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.377+0000 7fcf4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:16.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.377+0000 7fcf617fa640 4 set_mon_vals no callback set 2024-03-25T18:13:16.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.377+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcf58003860 con 0x7fcf6413f710 2024-03-25T18:13:16.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.377+0000 7fcf4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:16.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.377+0000 7fcf4b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:16.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.377+0000 7fcf4b7fe640 10 monclient: dump: 2024-03-25T18:13:16.379 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:16.379 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:16.379 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:16.379 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:16.379 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:16.379 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:16.379 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:16.379 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:16.379 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:16.379 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:16.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.378+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fcf58003a80 con 0x7fcf6413f710 2024-03-25T18:13:16.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.378+0000 7fcf62ffd640 10 monclient: get_auth_request con 0x7fcf5006ddf0 auth_method 0 2024-03-25T18:13:16.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.378+0000 7fcf4b7fe640 10 client.4384.objecter ms_handle_connect 0x7fcf5006ddf0 2024-03-25T18:13:16.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.383+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fcf58004480 con 0x7fcf6413f710 2024-03-25T18:13:16.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.383+0000 7fcf4b7fe640 10 client.4384.objecter ms_dispatch 0x7fcf64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:16.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.383+0000 7fcf4b7fe640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:16.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.383+0000 7fcf4b7fe640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:16.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.383+0000 7fcf4b7fe640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-25T18:13:16.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.384+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcf58050210 con 0x7fcf6413f710 2024-03-25T18:13:16.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.384+0000 7fcf4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:16.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.384+0000 7fcf4b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:16.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.638+0000 7fcf69313640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:16.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.638+0000 7fcf69313640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:16.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.638+0000 7fcf69313640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:13:16.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.638+0000 7fcf69313640 1 -- 172.21.15.64:0/1888035775 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf640abb30 con 0x7fcf6413f710 2024-03-25T18:13:16.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.665+0000 7fcf4b7fe640 1 -- 172.21.15.64:0/1888035775 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fcf58052080 con 0x7fcf6413f710 2024-03-25T18:13:16.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.665+0000 7fcf4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:16.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.665+0000 7fcf4b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:16.666 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:13:16.666 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:16.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.668+0000 7fcf69313640 10 librados: watch_flush enter 2024-03-25T18:13:16.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.668+0000 7fcf69313640 10 librados: watch_flush exit 2024-03-25T18:13:16.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.668+0000 7fcf69313640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-25T18:13:16.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.668+0000 7fcf69313640 10 client.4384.objecter successfully canceled tick 2024-03-25T18:13:16.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.668+0000 7fcf69313640 1 -- 172.21.15.64:0/1888035775 >> v1:172.21.15.64:6800/2915543638 conn(0x7fcf5006ddf0 legacy=0x7fcf500702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:16.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.668+0000 7fcf69313640 10 monclient: shutdown 2024-03-25T18:13:16.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.668+0000 7fcf69313640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:16.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.668+0000 7fcf69313640 1 -- 172.21.15.64:0/1888035775 >> v1:172.21.15.86:6789/0 conn(0x7fcf6413f710 legacy=0x7fcf641d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:16.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.668+0000 7fcf69313640 1 -- 172.21.15.64:0/1888035775 shutdown_connections 2024-03-25T18:13:16.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.668+0000 7fcf69313640 1 -- 172.21.15.64:0/1888035775 >> 172.21.15.64:0/1888035775 conn(0x7fcf640b7ea0 msgr2=0x7fcf64074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:16.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.668+0000 7fcf69313640 1 -- 172.21.15.64:0/1888035775 shutdown_connections 2024-03-25T18:13:16.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.668+0000 7fcf69313640 1 -- 172.21.15.64:0/1888035775 wait complete. 2024-03-25T18:13:16.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:16.668+0000 7fcf69313640 1 librados: shutdown 2024-03-25T18:13:16.681 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:23.488 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:23.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.604+0000 7f2f939e9640 10 monclient: get_monmap_and_config 2024-03-25T18:13:23.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.604+0000 7f2f939e9640 10 monclient: build_initial_monmap 2024-03-25T18:13:23.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.604+0000 7f2f939e9640 10 monclient: monmap: 2024-03-25T18:13:23.606 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:13:23.606 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:23.606 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:13:23.605750+0000 2024-03-25T18:13:23.606 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:13:23.605750+0000 2024-03-25T18:13:23.606 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:23.606 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:23.606 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:13:23.606 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:13:23.606 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:13:23.606 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:23.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.605+0000 7f2f939e9640 1 Processor -- start 2024-03-25T18:13:23.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.606+0000 7f2f939e9640 1 -- start start 2024-03-25T18:13:23.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.606+0000 7f2f939e9640 10 monclient: init 2024-03-25T18:13:23.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.606+0000 7f2f939e9640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:23.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.606+0000 7f2f939e9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:23.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.606+0000 7f2f939e9640 10 monclient(hunting): picked mon.noname-b con 0x7f2f8c0a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:13:23.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.606+0000 7f2f939e9640 10 monclient(hunting): picked mon.noname-c con 0x7f2f8c13f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:13:23.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.606+0000 7f2f939e9640 10 monclient(hunting): picked mon.noname-a con 0x7f2f8c1430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:13:23.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.606+0000 7f2f939e9640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c142520 con 0x7f2f8c1430c0 2024-03-25T18:13:23.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.606+0000 7f2f939e9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c0b28a0 con 0x7f2f8c13f710 2024-03-25T18:13:23.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.606+0000 7f2f939e9640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c0b2a30 con 0x7f2f8c0a98f0 2024-03-25T18:13:23.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.606+0000 7f2f939e9640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:23.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.606+0000 7f2f939e9640 10 monclient(hunting): authenticate will time out at 7189.845382s 2024-03-25T18:13:23.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.617+0000 7f2f91f86640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f2f8c1430c0 0x7f2f8c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:54314/0 (socket says 172.21.15.64:54314) 2024-03-25T18:13:23.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.617+0000 7f2f91f86640 1 -- 172.21.15.64:0/1783911516 learned_addr learned my addr 172.21.15.64:0/1783911516 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:13:23.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.622+0000 7f2f83fff640 1 -- 172.21.15.64:0/1783911516 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155607049 0 0) 0x7f2f8c142520 con 0x7f2f8c1430c0 2024-03-25T18:13:23.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.622+0000 7f2f83fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:23.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.622+0000 7f2f83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:23.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.622+0000 7f2f83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:23.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.622+0000 7f2f83fff640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:13:23.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.622+0000 7f2f83fff640 1 -- 172.21.15.64:0/1783911516 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f78003520 con 0x7f2f8c1430c0 2024-03-25T18:13:23.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.634+0000 7f2f83fff640 1 -- 172.21.15.64:0/1783911516 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1285904314 0 0) 0x7f2f78003520 con 0x7f2f8c1430c0 2024-03-25T18:13:23.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.634+0000 7f2f83fff640 1 -- 172.21.15.64:0/1783911516 >> v1:172.21.15.64:6790/0 conn(0x7f2f8c0a98f0 legacy=0x7f2f8c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:13:23.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.634+0000 7f2f83fff640 1 -- 172.21.15.64:0/1783911516 >> v1:172.21.15.86:6789/0 conn(0x7f2f8c13f710 legacy=0x7f2f8c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:23.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.634+0000 7f2f83fff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:23.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.634+0000 7f2f83fff640 1 monclient: found mon.noname-a 2024-03-25T18:13:23.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.634+0000 7f2f83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:23.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.634+0000 7f2f83fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:13:23.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.634+0000 7f2f83fff640 1 -- 172.21.15.64:0/1783911516 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8c0b44c0 con 0x7f2f8c1430c0 2024-03-25T18:13:23.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.634+0000 7f2f83fff640 10 monclient: _finish_auth 0 2024-03-25T18:13:23.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.634+0000 7f2f83fff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:23.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.634+0000 7f2f83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:23.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.634+0000 7f2f83fff640 1 -- 172.21.15.64:0/1783911516 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2f84001a10 con 0x7f2f8c1430c0 2024-03-25T18:13:23.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.634+0000 7f2f939e9640 5 monclient: authenticate success, global_id 4392 2024-03-25T18:13:23.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.634+0000 7f2f939e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:23.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.634+0000 7f2f83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:23.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.634+0000 7f2f83fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:23.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.634+0000 7f2f83fff640 10 monclient: dump: 2024-03-25T18:13:23.636 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:23.636 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:23.636 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:23.636 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:23.637 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:23.637 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:23.637 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:23.637 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:23.637 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:23.637 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:23.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.634+0000 7f2f939e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:23.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.640+0000 7f2f83fff640 1 -- 172.21.15.64:0/1783911516 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f84001be0 con 0x7f2f8c1430c0 2024-03-25T18:13:23.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.640+0000 7f2f83fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:23.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.640+0000 7f2f83fff640 1 -- 172.21.15.64:0/1783911516 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2f84002030 con 0x7f2f8c1430c0 2024-03-25T18:13:23.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.640+0000 7f2f83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:23.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.640+0000 7f2f83fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:23.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.640+0000 7f2f83fff640 10 monclient: dump: 2024-03-25T18:13:23.642 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:23.642 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:23.642 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:23.642 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:23.642 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:23.642 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:23.642 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:23.642 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:23.642 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:23.642 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:23.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.640+0000 7f2f939e9640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:23.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.640+0000 7f2f939e9640 4 set_mon_vals no callback set 2024-03-25T18:13:23.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.641+0000 7f2f939e9640 10 monclient: shutdown 2024-03-25T18:13:23.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.641+0000 7f2f939e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:23.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.641+0000 7f2f939e9640 1 -- 172.21.15.64:0/1783911516 >> v1:172.21.15.64:6789/0 conn(0x7f2f8c1430c0 legacy=0x7f2f8c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:23.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.641+0000 7f2f939e9640 1 -- 172.21.15.64:0/1783911516 shutdown_connections 2024-03-25T18:13:23.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.641+0000 7f2f939e9640 1 -- 172.21.15.64:0/1783911516 >> 172.21.15.64:0/1783911516 conn(0x7f2f8c0b7ea0 msgr2=0x7f2f8c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:23.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.641+0000 7f2f939e9640 1 -- 172.21.15.64:0/1783911516 shutdown_connections 2024-03-25T18:13:23.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.641+0000 7f2f939e9640 1 -- 172.21.15.64:0/1783911516 wait complete. 2024-03-25T18:13:23.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.642+0000 7f2f939e9640 10 monclient: build_initial_monmap 2024-03-25T18:13:23.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.642+0000 7f2f939e9640 10 monclient: monmap: 2024-03-25T18:13:23.645 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:13:23.645 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:23.645 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:13:23.645 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:13:23.645 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:23.645 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:13:23.645 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:13:23.645 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:13:23.645 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:13:23.645 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:23.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.643+0000 7f2f939e9640 1 librados: starting msgr at 2024-03-25T18:13:23.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.643+0000 7f2f939e9640 1 librados: starting objecter 2024-03-25T18:13:23.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.643+0000 7f2f939e9640 1 Processor -- start 2024-03-25T18:13:23.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.643+0000 7f2f939e9640 1 -- start start 2024-03-25T18:13:23.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.643+0000 7f2f939e9640 1 librados: setting wanted keys 2024-03-25T18:13:23.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.643+0000 7f2f939e9640 1 librados: calling monclient init 2024-03-25T18:13:23.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.643+0000 7f2f939e9640 10 monclient: init 2024-03-25T18:13:23.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.643+0000 7f2f939e9640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:23.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.643+0000 7f2f939e9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:23.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.643+0000 7f2f939e9640 10 monclient(hunting): picked mon.noname-b con 0x7f2f8c0a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:13:23.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.643+0000 7f2f939e9640 10 monclient(hunting): picked mon.noname-c con 0x7f2f8c13f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:13:23.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.643+0000 7f2f939e9640 10 monclient(hunting): picked mon.noname-a con 0x7f2f8c1430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:13:23.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.643+0000 7f2f939e9640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c0abde0 con 0x7f2f8c1430c0 2024-03-25T18:13:23.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.643+0000 7f2f939e9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c1d51f0 con 0x7f2f8c0a98f0 2024-03-25T18:13:23.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.643+0000 7f2f939e9640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c1d83d0 con 0x7f2f8c13f710 2024-03-25T18:13:23.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.643+0000 7f2f939e9640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:23.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.643+0000 7f2f939e9640 10 monclient(hunting): authenticate will time out at 7189.882354s 2024-03-25T18:13:23.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.647+0000 7f2f91f86640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f2f8c1430c0 0x7f2f8c1e1dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:54358/0 (socket says 172.21.15.64:54358) 2024-03-25T18:13:23.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.647+0000 7f2f91f86640 1 -- 172.21.15.64:0/2084071228 learned_addr learned my addr 172.21.15.64:0/2084071228 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:13:23.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.651+0000 7f2f81ffb640 10 client.?.objecter ms_handle_connect 0x7f2f8c1430c0 2024-03-25T18:13:23.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.652+0000 7f2f81ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:23.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.663+0000 7f2f81ffb640 10 client.?.objecter ms_handle_connect 0x7f2f8c0a98f0 2024-03-25T18:13:23.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.663+0000 7f2f81ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:23.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.664+0000 7f2f81ffb640 1 -- 172.21.15.64:0/2084071228 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487944433 0 0) 0x7f2f8c0abde0 con 0x7f2f8c1430c0 2024-03-25T18:13:23.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.664+0000 7f2f81ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:23.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.664+0000 7f2f81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:23.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.664+0000 7f2f81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:23.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.664+0000 7f2f81ffb640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:13:23.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.664+0000 7f2f81ffb640 1 -- 172.21.15.64:0/2084071228 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f70004eb0 con 0x7f2f8c1430c0 2024-03-25T18:13:23.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.676+0000 7f2f81ffb640 1 -- 172.21.15.64:0/2084071228 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2005344396 0 0) 0x7f2f70004eb0 con 0x7f2f8c1430c0 2024-03-25T18:13:23.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.676+0000 7f2f81ffb640 1 -- 172.21.15.64:0/2084071228 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f8c0abde0 con 0x7f2f8c1430c0 2024-03-25T18:13:23.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.676+0000 7f2f81ffb640 1 -- 172.21.15.64:0/2084071228 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2f84001ef0 con 0x7f2f8c1430c0 2024-03-25T18:13:23.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.676+0000 7f2f81ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:23.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.676+0000 7f2f81ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:23.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.676+0000 7f2f81ffb640 10 monclient(hunting): dump: 2024-03-25T18:13:23.677 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:23.677 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:23.677 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:23.678 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:23.678 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:23.678 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:23.678 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:23.678 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:23.678 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:23.678 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:23.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.676+0000 7f2f81ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:23.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.676+0000 7f2f81ffb640 10 client.?.objecter ms_handle_connect 0x7f2f8c13f710 2024-03-25T18:13:23.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.676+0000 7f2f81ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:23.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.684+0000 7f2f81ffb640 1 -- 172.21.15.64:0/2084071228 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3832638366 0 0) 0x7f2f8c0abde0 con 0x7f2f8c1430c0 2024-03-25T18:13:23.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.684+0000 7f2f81ffb640 1 -- 172.21.15.64:0/2084071228 >> v1:172.21.15.64:6790/0 conn(0x7f2f8c13f710 legacy=0x7f2f8c1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:23.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.684+0000 7f2f81ffb640 1 -- 172.21.15.64:0/2084071228 >> v1:172.21.15.86:6789/0 conn(0x7f2f8c0a98f0 legacy=0x7f2f8c138830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:23.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.684+0000 7f2f81ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:23.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.684+0000 7f2f81ffb640 1 monclient: found mon.a 2024-03-25T18:13:23.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.684+0000 7f2f81ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:23.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.684+0000 7f2f81ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:13:23.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.684+0000 7f2f81ffb640 1 -- 172.21.15.64:0/2084071228 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8c1d77a0 con 0x7f2f8c1430c0 2024-03-25T18:13:23.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.684+0000 7f2f81ffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:23.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.684+0000 7f2f81ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:23.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.684+0000 7f2f81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:23.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.684+0000 7f2f939e9640 5 monclient: authenticate success, global_id 4395 2024-03-25T18:13:23.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.684+0000 7f2f939e9640 10 monclient: _renew_subs 2024-03-25T18:13:23.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.684+0000 7f2f939e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:13:23.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.684+0000 7f2f939e9640 1 -- 172.21.15.64:0/2084071228 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8c137f40 con 0x7f2f8c1430c0 2024-03-25T18:13:23.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.685+0000 7f2f939e9640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:23.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.685+0000 7f2f939e9640 10 monclient: _renew_subs 2024-03-25T18:13:23.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.685+0000 7f2f939e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:13:23.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.685+0000 7f2f939e9640 1 -- 172.21.15.64:0/2084071228 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f8c1383d0 con 0x7f2f8c1430c0 2024-03-25T18:13:23.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.685+0000 7f2f939e9640 1 librados: init done 2024-03-25T18:13:23.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.686+0000 7f2f939e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:23.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.686+0000 7f2f939e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:23.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.686+0000 7f2f939e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:13:23.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.686+0000 7f2f939e9640 1 -- 172.21.15.64:0/2084071228 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8c1d79e0 con 0x7f2f8c1430c0 2024-03-25T18:13:23.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.989+0000 7f2f81ffb640 1 -- 172.21.15.64:0/2084071228 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f84004e10 con 0x7f2f8c1430c0 2024-03-25T18:13:23.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.989+0000 7f2f81ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:23.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.989+0000 7f2f81ffb640 1 -- 172.21.15.64:0/2084071228 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2f84004f80 con 0x7f2f8c1430c0 2024-03-25T18:13:23.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.989+0000 7f2f81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:23.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.989+0000 7f2f81ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:23.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.989+0000 7f2f81ffb640 10 monclient: dump: 2024-03-25T18:13:23.990 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:23.990 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:23.991 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:23.991 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:23.991 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:23.991 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:23.991 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:23.991 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:23.991 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:23.991 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:23.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.989+0000 7f2f837fe640 4 set_mon_vals no callback set 2024-03-25T18:13:23.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.990+0000 7f2f81ffb640 1 -- 172.21.15.64:0/2084071228 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f2f8401af50 con 0x7f2f8c1430c0 2024-03-25T18:13:23.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.990+0000 7f2f91785640 10 monclient: get_auth_request con 0x7f2f7006dd20 auth_method 0 2024-03-25T18:13:23.992 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.990+0000 7f2f81ffb640 10 client.4395.objecter ms_handle_connect 0x7f2f7006dd20 2024-03-25T18:13:23.997 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.997+0000 7f2f81ffb640 1 -- 172.21.15.64:0/2084071228 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f2f84054a90 con 0x7f2f8c1430c0 2024-03-25T18:13:23.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.997+0000 7f2f81ffb640 10 client.4395.objecter ms_dispatch 0x7f2f8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:23.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.997+0000 7f2f81ffb640 3 client.4395.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:23.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.997+0000 7f2f81ffb640 3 client.4395.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:23.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.997+0000 7f2f81ffb640 20 client.4395.objecter dump_active .. 0 homeless 2024-03-25T18:13:23.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.997+0000 7f2f81ffb640 1 -- 172.21.15.64:0/2084071228 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f840505a0 con 0x7f2f8c1430c0 2024-03-25T18:13:23.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.997+0000 7f2f81ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:23.998 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:23.997+0000 7f2f81ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:24.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.241+0000 7f2f939e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:24.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.241+0000 7f2f939e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:24.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.241+0000 7f2f939e9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:13:24.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.241+0000 7f2f939e9640 1 -- 172.21.15.64:0/2084071228 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f8c1d7590 con 0x7f2f8c1430c0 2024-03-25T18:13:24.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.280+0000 7f2f81ffb640 1 -- 172.21.15.64:0/2084071228 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f2f840524d0 con 0x7f2f8c1430c0 2024-03-25T18:13:24.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.280+0000 7f2f81ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:24.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.280+0000 7f2f81ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:24.281 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:13:24.281 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:24.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.282+0000 7f2f939e9640 10 librados: watch_flush enter 2024-03-25T18:13:24.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.282+0000 7f2f939e9640 10 librados: watch_flush exit 2024-03-25T18:13:24.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.282+0000 7f2f939e9640 20 client.4395.objecter shutdown clearing up homeless session... 2024-03-25T18:13:24.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.282+0000 7f2f939e9640 10 client.4395.objecter successfully canceled tick 2024-03-25T18:13:24.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.282+0000 7f2f939e9640 1 -- 172.21.15.64:0/2084071228 >> v1:172.21.15.64:6800/2915543638 conn(0x7f2f7006dd20 legacy=0x7f2f700701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:24.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.282+0000 7f2f939e9640 10 monclient: shutdown 2024-03-25T18:13:24.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.282+0000 7f2f939e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:24.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.282+0000 7f2f939e9640 1 -- 172.21.15.64:0/2084071228 >> v1:172.21.15.64:6789/0 conn(0x7f2f8c1430c0 legacy=0x7f2f8c1e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:24.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.282+0000 7f2f939e9640 1 -- 172.21.15.64:0/2084071228 shutdown_connections 2024-03-25T18:13:24.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.282+0000 7f2f939e9640 1 -- 172.21.15.64:0/2084071228 >> 172.21.15.64:0/2084071228 conn(0x7f2f8c0b7ea0 msgr2=0x7f2f8c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:24.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.282+0000 7f2f939e9640 1 -- 172.21.15.64:0/2084071228 shutdown_connections 2024-03-25T18:13:24.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.282+0000 7f2f939e9640 1 -- 172.21.15.64:0/2084071228 wait complete. 2024-03-25T18:13:24.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:24.282+0000 7f2f939e9640 1 librados: shutdown 2024-03-25T18:13:24.294 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:31.100 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:31.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.215+0000 7f0b983c6640 10 monclient: get_monmap_and_config 2024-03-25T18:13:31.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.215+0000 7f0b983c6640 10 monclient: build_initial_monmap 2024-03-25T18:13:31.217 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.215+0000 7f0b983c6640 10 monclient: monmap: 2024-03-25T18:13:31.217 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:13:31.217 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:31.217 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:13:31.216951+0000 2024-03-25T18:13:31.217 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:13:31.216951+0000 2024-03-25T18:13:31.217 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:31.217 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:31.217 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:13:31.217 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:13:31.217 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:13:31.217 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:31.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.215+0000 7f0b983c6640 1 Processor -- start 2024-03-25T18:13:31.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.215+0000 7f0b983c6640 1 -- start start 2024-03-25T18:13:31.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.215+0000 7f0b983c6640 10 monclient: init 2024-03-25T18:13:31.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.216+0000 7f0b983c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:31.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.216+0000 7f0b983c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:31.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.216+0000 7f0b983c6640 10 monclient(hunting): picked mon.noname-a con 0x7f0b900b4d60 addr v1:172.21.15.64:6789/0 2024-03-25T18:13:31.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.216+0000 7f0b983c6640 10 monclient(hunting): picked mon.noname-b con 0x7f0b900b1730 addr v1:172.21.15.64:6790/0 2024-03-25T18:13:31.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.216+0000 7f0b983c6640 10 monclient(hunting): picked mon.noname-c con 0x7f0b9014be90 addr v1:172.21.15.86:6789/0 2024-03-25T18:13:31.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.216+0000 7f0b983c6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b900af940 con 0x7f0b900b4d60 2024-03-25T18:13:31.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.216+0000 7f0b983c6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b900b0a70 con 0x7f0b9014be90 2024-03-25T18:13:31.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.216+0000 7f0b983c6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b900b0d50 con 0x7f0b900b1730 2024-03-25T18:13:31.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.216+0000 7f0b983c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:31.219 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.216+0000 7f0b983c6640 10 monclient(hunting): authenticate will time out at 7197.456133s 2024-03-25T18:13:31.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.223+0000 7f0b96963640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f0b9014be90 0x7f0b9014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:47624/0 (socket says 172.21.15.64:47624) 2024-03-25T18:13:31.225 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.223+0000 7f0b96963640 1 -- 172.21.15.64:0/3603515362 learned_addr learned my addr 172.21.15.64:0/3603515362 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:13:31.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.238+0000 7f0b95160640 1 -- 172.21.15.64:0/3603515362 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969662726 0 0) 0x7f0b900af940 con 0x7f0b900b4d60 2024-03-25T18:13:31.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.238+0000 7f0b95160640 10 monclient(hunting): _negotiate 2024-03-25T18:13:31.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.238+0000 7f0b95160640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:31.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.238+0000 7f0b95160640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:31.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.238+0000 7f0b95160640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:13:31.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.238+0000 7f0b95160640 1 -- 172.21.15.64:0/3603515362 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b7c003520 con 0x7f0b900b4d60 2024-03-25T18:13:31.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.244+0000 7f0b95160640 1 -- 172.21.15.64:0/3603515362 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746064990 0 0) 0x7f0b900b0a70 con 0x7f0b9014be90 2024-03-25T18:13:31.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.244+0000 7f0b95160640 10 monclient(hunting): _negotiate 2024-03-25T18:13:31.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.244+0000 7f0b95160640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:31.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.244+0000 7f0b95160640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:31.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.244+0000 7f0b95160640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:13:31.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.244+0000 7f0b95160640 1 -- 172.21.15.64:0/3603515362 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b900af940 con 0x7f0b9014be90 2024-03-25T18:13:31.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.255+0000 7f0b95160640 1 -- 172.21.15.64:0/3603515362 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3353163251 0 0) 0x7f0b7c003520 con 0x7f0b900b4d60 2024-03-25T18:13:31.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.256+0000 7f0b95160640 1 -- 172.21.15.64:0/3603515362 >> v1:172.21.15.64:6790/0 conn(0x7f0b900b1730 legacy=0x7f0b900b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:31.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.256+0000 7f0b95160640 1 -- 172.21.15.64:0/3603515362 >> v1:172.21.15.86:6789/0 conn(0x7f0b9014be90 legacy=0x7f0b9014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:31.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.256+0000 7f0b95160640 10 monclient: _finish_hunting 0 2024-03-25T18:13:31.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.256+0000 7f0b95160640 1 monclient: found mon.noname-a 2024-03-25T18:13:31.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.256+0000 7f0b95160640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:31.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.256+0000 7f0b95160640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:13:31.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.256+0000 7f0b95160640 1 -- 172.21.15.64:0/3603515362 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b900b1030 con 0x7f0b900b4d60 2024-03-25T18:13:31.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.256+0000 7f0b95160640 10 monclient: _finish_auth 0 2024-03-25T18:13:31.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.256+0000 7f0b95160640 10 monclient: _check_auth_tickets 2024-03-25T18:13:31.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.256+0000 7f0b95160640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:31.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.256+0000 7f0b95160640 1 -- 172.21.15.64:0/3603515362 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0b8c001690 con 0x7f0b900b4d60 2024-03-25T18:13:31.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.256+0000 7f0b95160640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:31.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.256+0000 7f0b95160640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:31.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.256+0000 7f0b95160640 10 monclient: dump: 2024-03-25T18:13:31.259 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:31.259 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:31.259 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:31.259 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:31.259 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:31.259 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:31.259 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:31.259 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:31.259 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:31.259 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:31.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.256+0000 7f0b983c6640 5 monclient: authenticate success, global_id 4398 2024-03-25T18:13:31.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.256+0000 7f0b983c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:31.263 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.261+0000 7f0b95160640 1 -- 172.21.15.64:0/3603515362 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b8c0017f0 con 0x7f0b900b4d60 2024-03-25T18:13:31.263 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.261+0000 7f0b95160640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:31.263 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.261+0000 7f0b95160640 1 -- 172.21.15.64:0/3603515362 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0b8c001c00 con 0x7f0b900b4d60 2024-03-25T18:13:31.263 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.261+0000 7f0b95160640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:31.263 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.261+0000 7f0b95160640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:31.263 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.261+0000 7f0b95160640 10 monclient: dump: 2024-03-25T18:13:31.263 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:31.263 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:31.264 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:31.264 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:31.264 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:31.264 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:31.264 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:31.264 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:31.264 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:31.264 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:31.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.261+0000 7f0b983c6640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:31.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.261+0000 7f0b983c6640 4 set_mon_vals no callback set 2024-03-25T18:13:31.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.262+0000 7f0b983c6640 10 monclient: shutdown 2024-03-25T18:13:31.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.262+0000 7f0b983c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:31.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.262+0000 7f0b983c6640 1 -- 172.21.15.64:0/3603515362 >> v1:172.21.15.64:6789/0 conn(0x7f0b900b4d60 legacy=0x7f0b9013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:31.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.262+0000 7f0b983c6640 1 -- 172.21.15.64:0/3603515362 shutdown_connections 2024-03-25T18:13:31.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.262+0000 7f0b983c6640 1 -- 172.21.15.64:0/3603515362 >> 172.21.15.64:0/3603515362 conn(0x7f0b900617a0 msgr2=0x7f0b90061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:31.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.262+0000 7f0b983c6640 1 -- 172.21.15.64:0/3603515362 shutdown_connections 2024-03-25T18:13:31.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.262+0000 7f0b983c6640 1 -- 172.21.15.64:0/3603515362 wait complete. 2024-03-25T18:13:31.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.263+0000 7f0b983c6640 10 monclient: build_initial_monmap 2024-03-25T18:13:31.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.263+0000 7f0b983c6640 10 monclient: monmap: 2024-03-25T18:13:31.267 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:13:31.267 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:31.267 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:13:31.267 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:13:31.267 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:31.267 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:13:31.267 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:13:31.267 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:13:31.267 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:13:31.267 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:31.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.263+0000 7f0b983c6640 1 librados: starting msgr at 2024-03-25T18:13:31.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.263+0000 7f0b983c6640 1 librados: starting objecter 2024-03-25T18:13:31.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.263+0000 7f0b983c6640 1 Processor -- start 2024-03-25T18:13:31.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.263+0000 7f0b983c6640 1 -- start start 2024-03-25T18:13:31.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.263+0000 7f0b983c6640 1 librados: setting wanted keys 2024-03-25T18:13:31.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.263+0000 7f0b983c6640 1 librados: calling monclient init 2024-03-25T18:13:31.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.263+0000 7f0b983c6640 10 monclient: init 2024-03-25T18:13:31.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.264+0000 7f0b983c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:31.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.264+0000 7f0b983c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:31.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.264+0000 7f0b983c6640 10 monclient(hunting): picked mon.noname-b con 0x7f0b900b1730 addr v1:172.21.15.86:6789/0 2024-03-25T18:13:31.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.264+0000 7f0b983c6640 10 monclient(hunting): picked mon.noname-c con 0x7f0b900b4d60 addr v1:172.21.15.64:6790/0 2024-03-25T18:13:31.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.264+0000 7f0b983c6640 10 monclient(hunting): picked mon.noname-a con 0x7f0b9014be90 addr v1:172.21.15.64:6789/0 2024-03-25T18:13:31.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.264+0000 7f0b983c6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b901d79d0 con 0x7f0b9014be90 2024-03-25T18:13:31.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.264+0000 7f0b983c6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b90137100 con 0x7f0b900b1730 2024-03-25T18:13:31.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.264+0000 7f0b983c6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b90137290 con 0x7f0b900b4d60 2024-03-25T18:13:31.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.264+0000 7f0b983c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:31.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.264+0000 7f0b983c6640 10 monclient(hunting): authenticate will time out at 7197.504319s 2024-03-25T18:13:31.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.277+0000 7f0b96963640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f0b9014be90 0x7f0b9013b270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:41440/0 (socket says 172.21.15.64:41440) 2024-03-25T18:13:31.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.277+0000 7f0b96963640 1 -- 172.21.15.64:0/263957995 learned_addr learned my addr 172.21.15.64:0/263957995 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:13:31.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.278+0000 7f0b86ffd640 10 client.?.objecter ms_handle_connect 0x7f0b9014be90 2024-03-25T18:13:31.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.278+0000 7f0b86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:31.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.282+0000 7f0b86ffd640 10 client.?.objecter ms_handle_connect 0x7f0b900b1730 2024-03-25T18:13:31.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.282+0000 7f0b86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:31.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.283+0000 7f0b86ffd640 10 client.?.objecter ms_handle_connect 0x7f0b900b4d60 2024-03-25T18:13:31.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.283+0000 7f0b86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:31.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.284+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978256381 0 0) 0x7f0b901d79d0 con 0x7f0b9014be90 2024-03-25T18:13:31.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.284+0000 7f0b86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:31.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.284+0000 7f0b86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:31.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.284+0000 7f0b86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:31.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.284+0000 7f0b86ffd640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:13:31.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.284+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b74004eb0 con 0x7f0b9014be90 2024-03-25T18:13:31.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.295+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198439527 0 0) 0x7f0b90137100 con 0x7f0b900b1730 2024-03-25T18:13:31.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.295+0000 7f0b86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:31.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.295+0000 7f0b86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:31.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.295+0000 7f0b86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:31.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.295+0000 7f0b86ffd640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:13:31.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.295+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b901d79d0 con 0x7f0b900b1730 2024-03-25T18:13:31.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.302+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1847672446 0 0) 0x7f0b74004eb0 con 0x7f0b9014be90 2024-03-25T18:13:31.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.302+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b90137100 con 0x7f0b9014be90 2024-03-25T18:13:31.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.302+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0b88001c10 con 0x7f0b9014be90 2024-03-25T18:13:31.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.302+0000 7f0b86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:31.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.302+0000 7f0b86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:31.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.302+0000 7f0b86ffd640 10 monclient(hunting): dump: 2024-03-25T18:13:31.304 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:31.304 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:31.305 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:31.305 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:31.305 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:31.305 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:31.305 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:31.305 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:31.305 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:31.305 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:31.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.302+0000 7f0b86ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:31.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.313+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497827659 0 0) 0x7f0b90137290 con 0x7f0b900b4d60 2024-03-25T18:13:31.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.313+0000 7f0b86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:31.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.313+0000 7f0b86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:31.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.313+0000 7f0b86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:31.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.313+0000 7f0b86ffd640 10 monclient(hunting): my global_id is 4403 2024-03-25T18:13:31.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.313+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b74004eb0 con 0x7f0b900b4d60 2024-03-25T18:13:31.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.317+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2648360073 0 0) 0x7f0b901d79d0 con 0x7f0b900b1730 2024-03-25T18:13:31.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.317+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b90137290 con 0x7f0b900b1730 2024-03-25T18:13:31.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.317+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0b8c001b00 con 0x7f0b900b1730 2024-03-25T18:13:31.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.317+0000 7f0b86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:31.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.317+0000 7f0b86ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:31.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.317+0000 7f0b86ffd640 10 monclient(hunting): dump: 2024-03-25T18:13:31.319 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:31.319 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:31.319 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:31.320 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:31.320 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:31.320 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:31.320 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:31.320 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:31.320 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:31.320 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:31.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4168436311 0 0) 0x7f0b90137100 con 0x7f0b9014be90 2024-03-25T18:13:31.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 >> v1:172.21.15.64:6790/0 conn(0x7f0b900b4d60 legacy=0x7f0b901d7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:31.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 >> v1:172.21.15.86:6789/0 conn(0x7f0b900b1730 legacy=0x7f0b901369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:31.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b86ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:13:31.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b86ffd640 1 monclient: found mon.a 2024-03-25T18:13:31.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:31.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b86ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:13:31.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b901d8600 con 0x7f0b9014be90 2024-03-25T18:13:31.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b86ffd640 10 monclient: _finish_auth 0 2024-03-25T18:13:31.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b86ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:13:31.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:31.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b983c6640 5 monclient: authenticate success, global_id 4401 2024-03-25T18:13:31.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b983c6640 10 monclient: _renew_subs 2024-03-25T18:13:31.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b983c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:13:31.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b983c6640 1 -- 172.21.15.64:0/263957995 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b90143900 con 0x7f0b9014be90 2024-03-25T18:13:31.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b983c6640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:31.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b983c6640 10 monclient: _renew_subs 2024-03-25T18:13:31.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b983c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:13:31.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b983c6640 1 -- 172.21.15.64:0/263957995 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b90135e30 con 0x7f0b9014be90 2024-03-25T18:13:31.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.332+0000 7f0b983c6640 1 librados: init done 2024-03-25T18:13:31.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.333+0000 7f0b983c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:31.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.333+0000 7f0b983c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:31.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.333+0000 7f0b983c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:13:31.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.333+0000 7f0b983c6640 1 -- 172.21.15.64:0/263957995 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b901d7b60 con 0x7f0b9014be90 2024-03-25T18:13:31.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.669+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b88004b60 con 0x7f0b9014be90 2024-03-25T18:13:31.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.669+0000 7f0b86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:31.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.669+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0b88004e70 con 0x7f0b9014be90 2024-03-25T18:13:31.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.669+0000 7f0b86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:31.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.669+0000 7f0b86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:31.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.669+0000 7f0b86ffd640 10 monclient: dump: 2024-03-25T18:13:31.672 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:31.672 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:31.672 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:31.672 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:31.672 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:31.672 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:31.672 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:31.672 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:31.672 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:31.672 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:31.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.669+0000 7f0b9495f640 4 set_mon_vals no callback set 2024-03-25T18:13:31.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.670+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f0b8801ae40 con 0x7f0b9014be90 2024-03-25T18:13:31.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.671+0000 7f0b96162640 10 monclient: get_auth_request con 0x7f0b7406df40 auth_method 0 2024-03-25T18:13:31.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.671+0000 7f0b86ffd640 10 client.4401.objecter ms_handle_connect 0x7f0b7406df40 2024-03-25T18:13:31.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.676+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f0b88054a80 con 0x7f0b9014be90 2024-03-25T18:13:31.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.676+0000 7f0b86ffd640 10 client.4401.objecter ms_dispatch 0x7f0b90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:31.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.676+0000 7f0b86ffd640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:31.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.676+0000 7f0b86ffd640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:31.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.676+0000 7f0b86ffd640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-25T18:13:31.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.676+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b880503c0 con 0x7f0b9014be90 2024-03-25T18:13:31.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.676+0000 7f0b86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:31.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.676+0000 7f0b86ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:31.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.929+0000 7f0b983c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:31.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.929+0000 7f0b983c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:31.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.929+0000 7f0b983c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:13:31.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:31.929+0000 7f0b983c6640 1 -- 172.21.15.64:0/263957995 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b90138390 con 0x7f0b9014be90 2024-03-25T18:13:32.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:32.086+0000 7f0b86ffd640 1 -- 172.21.15.64:0/263957995 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f0b880524c0 con 0x7f0b9014be90 2024-03-25T18:13:32.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:32.086+0000 7f0b86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:32.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:32.086+0000 7f0b86ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:32.088 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:13:32.088 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:32.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:32.088+0000 7f0b983c6640 10 librados: watch_flush enter 2024-03-25T18:13:32.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:32.088+0000 7f0b983c6640 10 librados: watch_flush exit 2024-03-25T18:13:32.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:32.088+0000 7f0b983c6640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-25T18:13:32.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:32.088+0000 7f0b983c6640 10 client.4401.objecter successfully canceled tick 2024-03-25T18:13:32.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:32.088+0000 7f0b983c6640 1 -- 172.21.15.64:0/263957995 >> v1:172.21.15.64:6800/2915543638 conn(0x7f0b7406df40 legacy=0x7f0b74070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:32.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:32.088+0000 7f0b983c6640 10 monclient: shutdown 2024-03-25T18:13:32.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:32.088+0000 7f0b983c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:32.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:32.088+0000 7f0b983c6640 1 -- 172.21.15.64:0/263957995 >> v1:172.21.15.64:6789/0 conn(0x7f0b9014be90 legacy=0x7f0b9013b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:32.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:32.088+0000 7f0b983c6640 1 -- 172.21.15.64:0/263957995 shutdown_connections 2024-03-25T18:13:32.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:32.088+0000 7f0b983c6640 1 -- 172.21.15.64:0/263957995 >> 172.21.15.64:0/263957995 conn(0x7f0b900617a0 msgr2=0x7f0b900b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:32.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:32.088+0000 7f0b983c6640 1 -- 172.21.15.64:0/263957995 shutdown_connections 2024-03-25T18:13:32.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:32.088+0000 7f0b983c6640 1 -- 172.21.15.64:0/263957995 wait complete. 2024-03-25T18:13:32.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:32.088+0000 7f0b983c6640 1 librados: shutdown 2024-03-25T18:13:32.101 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:38.906 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:39.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.020+0000 7fcf3a8a8640 10 monclient: get_monmap_and_config 2024-03-25T18:13:39.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.020+0000 7fcf3a8a8640 10 monclient: build_initial_monmap 2024-03-25T18:13:39.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.020+0000 7fcf3a8a8640 10 monclient: monmap: 2024-03-25T18:13:39.021 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:13:39.021 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:39.021 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:13:39.021332+0000 2024-03-25T18:13:39.021 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:13:39.021332+0000 2024-03-25T18:13:39.021 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:39.021 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:39.021 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:13:39.022 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:13:39.022 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:13:39.022 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:39.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.021+0000 7fcf3a8a8640 1 Processor -- start 2024-03-25T18:13:39.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.021+0000 7fcf3a8a8640 1 -- start start 2024-03-25T18:13:39.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.021+0000 7fcf3a8a8640 10 monclient: init 2024-03-25T18:13:39.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.021+0000 7fcf3a8a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:39.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.021+0000 7fcf3a8a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:39.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.021+0000 7fcf3a8a8640 10 monclient(hunting): picked mon.noname-c con 0x7fcf340a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:13:39.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.021+0000 7fcf3a8a8640 10 monclient(hunting): picked mon.noname-b con 0x7fcf3413f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:13:39.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.021+0000 7fcf3a8a8640 10 monclient(hunting): picked mon.noname-a con 0x7fcf341430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:13:39.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.021+0000 7fcf3a8a8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf34142520 con 0x7fcf341430c0 2024-03-25T18:13:39.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.021+0000 7fcf3a8a8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf340b28a0 con 0x7fcf340a98f0 2024-03-25T18:13:39.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.021+0000 7fcf3a8a8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf340b2a30 con 0x7fcf3413f710 2024-03-25T18:13:39.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.021+0000 7fcf3a8a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:39.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.021+0000 7fcf3a8a8640 10 monclient(hunting): authenticate will time out at 7205.260333s 2024-03-25T18:13:39.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.030+0000 7fcf33fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fcf340a98f0 0x7fcf340a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:49026/0 (socket says 172.21.15.64:49026) 2024-03-25T18:13:39.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.030+0000 7fcf33fff640 1 -- 172.21.15.64:0/3875524327 learned_addr learned my addr 172.21.15.64:0/3875524327 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:13:39.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.044+0000 7fcf32ffd640 1 -- 172.21.15.64:0/3875524327 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662071768 0 0) 0x7fcf340b28a0 con 0x7fcf340a98f0 2024-03-25T18:13:39.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.044+0000 7fcf32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:39.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.044+0000 7fcf32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:39.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.044+0000 7fcf32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:39.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.044+0000 7fcf32ffd640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:13:39.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.044+0000 7fcf32ffd640 1 -- 172.21.15.64:0/3875524327 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf1c003520 con 0x7fcf340a98f0 2024-03-25T18:13:39.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.050+0000 7fcf32ffd640 1 -- 172.21.15.64:0/3875524327 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766213988 0 0) 0x7fcf340b2a30 con 0x7fcf3413f710 2024-03-25T18:13:39.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.050+0000 7fcf32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:39.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.050+0000 7fcf32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:39.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.050+0000 7fcf32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:39.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.050+0000 7fcf32ffd640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:13:39.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.050+0000 7fcf32ffd640 1 -- 172.21.15.64:0/3875524327 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf340b28a0 con 0x7fcf3413f710 2024-03-25T18:13:39.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.057+0000 7fcf32ffd640 1 -- 172.21.15.64:0/3875524327 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1072960649 0 0) 0x7fcf1c003520 con 0x7fcf340a98f0 2024-03-25T18:13:39.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.057+0000 7fcf32ffd640 1 -- 172.21.15.64:0/3875524327 >> v1:172.21.15.64:6790/0 conn(0x7fcf3413f710 legacy=0x7fcf341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:39.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.057+0000 7fcf32ffd640 1 -- 172.21.15.64:0/3875524327 >> v1:172.21.15.64:6789/0 conn(0x7fcf341430c0 legacy=0x7fcf3414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:39.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.057+0000 7fcf32ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:13:39.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.057+0000 7fcf32ffd640 1 monclient: found mon.noname-c 2024-03-25T18:13:39.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.057+0000 7fcf32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:39.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.057+0000 7fcf32ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:13:39.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.057+0000 7fcf32ffd640 1 -- 172.21.15.64:0/3875524327 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf340b44c0 con 0x7fcf340a98f0 2024-03-25T18:13:39.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.057+0000 7fcf32ffd640 10 monclient: _finish_auth 0 2024-03-25T18:13:39.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.057+0000 7fcf32ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:13:39.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.057+0000 7fcf32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:39.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.057+0000 7fcf32ffd640 1 -- 172.21.15.64:0/3875524327 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcf24001690 con 0x7fcf340a98f0 2024-03-25T18:13:39.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.057+0000 7fcf3a8a8640 5 monclient: authenticate success, global_id 4399 2024-03-25T18:13:39.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.057+0000 7fcf3a8a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:39.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.057+0000 7fcf32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:39.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.057+0000 7fcf32ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:39.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.057+0000 7fcf32ffd640 10 monclient: dump: 2024-03-25T18:13:39.060 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:39.060 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:39.060 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:39.060 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:39.060 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:39.060 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:39.060 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:39.060 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:39.060 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:39.060 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:39.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.057+0000 7fcf3a8a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:39.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.061+0000 7fcf32ffd640 1 -- 172.21.15.64:0/3875524327 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf240017f0 con 0x7fcf340a98f0 2024-03-25T18:13:39.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.061+0000 7fcf32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:39.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.061+0000 7fcf32ffd640 1 -- 172.21.15.64:0/3875524327 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcf24001c00 con 0x7fcf340a98f0 2024-03-25T18:13:39.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.061+0000 7fcf3a8a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:39.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.061+0000 7fcf3a8a8640 4 set_mon_vals no callback set 2024-03-25T18:13:39.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.061+0000 7fcf32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:39.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.061+0000 7fcf32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:39.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.061+0000 7fcf32ffd640 10 monclient: dump: 2024-03-25T18:13:39.062 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:39.062 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:39.062 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:39.062 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:39.062 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:39.062 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:39.062 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:39.062 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:39.062 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:39.063 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:39.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.062+0000 7fcf3a8a8640 10 monclient: shutdown 2024-03-25T18:13:39.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.062+0000 7fcf3a8a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:39.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.062+0000 7fcf3a8a8640 1 -- 172.21.15.64:0/3875524327 >> v1:172.21.15.86:6789/0 conn(0x7fcf340a98f0 legacy=0x7fcf340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:39.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.062+0000 7fcf3a8a8640 1 -- 172.21.15.64:0/3875524327 shutdown_connections 2024-03-25T18:13:39.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.062+0000 7fcf3a8a8640 1 -- 172.21.15.64:0/3875524327 >> 172.21.15.64:0/3875524327 conn(0x7fcf340b7ea0 msgr2=0x7fcf340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:39.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.062+0000 7fcf3a8a8640 1 -- 172.21.15.64:0/3875524327 shutdown_connections 2024-03-25T18:13:39.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.062+0000 7fcf3a8a8640 1 -- 172.21.15.64:0/3875524327 wait complete. 2024-03-25T18:13:39.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.063+0000 7fcf3a8a8640 10 monclient: build_initial_monmap 2024-03-25T18:13:39.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.063+0000 7fcf3a8a8640 10 monclient: monmap: 2024-03-25T18:13:39.064 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:13:39.065 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:39.065 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:13:39.065 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:13:39.065 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:39.065 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:13:39.065 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:13:39.065 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:13:39.065 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:13:39.065 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:39.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.063+0000 7fcf3a8a8640 1 librados: starting msgr at 2024-03-25T18:13:39.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.063+0000 7fcf3a8a8640 1 librados: starting objecter 2024-03-25T18:13:39.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.063+0000 7fcf3a8a8640 1 Processor -- start 2024-03-25T18:13:39.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.063+0000 7fcf3a8a8640 1 -- start start 2024-03-25T18:13:39.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.063+0000 7fcf3a8a8640 1 librados: setting wanted keys 2024-03-25T18:13:39.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.063+0000 7fcf3a8a8640 1 librados: calling monclient init 2024-03-25T18:13:39.066 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.063+0000 7fcf3a8a8640 10 monclient: init 2024-03-25T18:13:39.066 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.064+0000 7fcf3a8a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:39.066 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.064+0000 7fcf3a8a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:39.066 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.064+0000 7fcf3a8a8640 10 monclient(hunting): picked mon.noname-c con 0x7fcf340a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:13:39.066 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.064+0000 7fcf3a8a8640 10 monclient(hunting): picked mon.noname-b con 0x7fcf3413f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:13:39.066 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.064+0000 7fcf3a8a8640 10 monclient(hunting): picked mon.noname-a con 0x7fcf341430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:13:39.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.064+0000 7fcf3a8a8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3406f000 con 0x7fcf341430c0 2024-03-25T18:13:39.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.064+0000 7fcf3a8a8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3406ff30 con 0x7fcf3413f710 2024-03-25T18:13:39.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.064+0000 7fcf3a8a8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf3406dea0 con 0x7fcf340a98f0 2024-03-25T18:13:39.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.064+0000 7fcf3a8a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:39.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.064+0000 7fcf3a8a8640 10 monclient(hunting): authenticate will time out at 7205.302913s 2024-03-25T18:13:39.078 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.078+0000 7fcf337fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fcf3413f710 0x7fcf34070930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:49052/0 (socket says 172.21.15.64:49052) 2024-03-25T18:13:39.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.078+0000 7fcf337fe640 1 -- 172.21.15.64:0/3099773413 learned_addr learned my addr 172.21.15.64:0/3099773413 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:13:39.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.079+0000 7fcf30ff9640 10 client.?.objecter ms_handle_connect 0x7fcf3413f710 2024-03-25T18:13:39.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.079+0000 7fcf30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:39.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.083+0000 7fcf30ff9640 10 client.?.objecter ms_handle_connect 0x7fcf340a98f0 2024-03-25T18:13:39.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.083+0000 7fcf30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:39.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.085+0000 7fcf30ff9640 10 client.?.objecter ms_handle_connect 0x7fcf341430c0 2024-03-25T18:13:39.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.085+0000 7fcf30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:39.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.091+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260145880 0 0) 0x7fcf3406ff30 con 0x7fcf3413f710 2024-03-25T18:13:39.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.091+0000 7fcf30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:39.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.091+0000 7fcf30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:39.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.091+0000 7fcf30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:39.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.091+0000 7fcf30ff9640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:13:39.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.091+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf20004eb0 con 0x7fcf3413f710 2024-03-25T18:13:39.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.100+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1676633515 0 0) 0x7fcf20004eb0 con 0x7fcf3413f710 2024-03-25T18:13:39.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.100+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf3406ff30 con 0x7fcf3413f710 2024-03-25T18:13:39.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.100+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcf28001a10 con 0x7fcf3413f710 2024-03-25T18:13:39.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.100+0000 7fcf30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:39.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.100+0000 7fcf30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:39.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.100+0000 7fcf30ff9640 10 monclient(hunting): dump: 2024-03-25T18:13:39.101 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:39.101 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:39.101 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:39.101 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:39.102 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:39.102 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:39.102 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:39.102 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:39.102 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:39.102 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:39.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.100+0000 7fcf30ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:39.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.110+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2679916963 0 0) 0x7fcf3406dea0 con 0x7fcf340a98f0 2024-03-25T18:13:39.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.110+0000 7fcf30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:39.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.110+0000 7fcf30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:39.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.110+0000 7fcf30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:39.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.110+0000 7fcf30ff9640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:13:39.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.110+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf20004eb0 con 0x7fcf340a98f0 2024-03-25T18:13:39.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.115+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114040905 0 0) 0x7fcf3406f000 con 0x7fcf341430c0 2024-03-25T18:13:39.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.115+0000 7fcf30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:39.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.115+0000 7fcf30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:39.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.115+0000 7fcf30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:39.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.115+0000 7fcf30ff9640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:13:39.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.115+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf3406dea0 con 0x7fcf341430c0 2024-03-25T18:13:39.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 951059873 0 0) 0x7fcf3406ff30 con 0x7fcf3413f710 2024-03-25T18:13:39.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 >> v1:172.21.15.64:6790/0 conn(0x7fcf340a98f0 legacy=0x7fcf3406e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:39.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 >> v1:172.21.15.64:6789/0 conn(0x7fcf341430c0 legacy=0x7fcf34071040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:39.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf30ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:13:39.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf30ff9640 1 monclient: found mon.b 2024-03-25T18:13:39.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:39.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf30ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:13:39.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf34139930 con 0x7fcf3413f710 2024-03-25T18:13:39.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf30ff9640 10 monclient: _finish_auth 0 2024-03-25T18:13:39.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf30ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:13:39.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:39.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf3a8a8640 5 monclient: authenticate success, global_id 4402 2024-03-25T18:13:39.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf3a8a8640 10 monclient: _renew_subs 2024-03-25T18:13:39.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf3a8a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:13:39.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf3a8a8640 1 -- 172.21.15.64:0/3099773413 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf341e76b0 con 0x7fcf3413f710 2024-03-25T18:13:39.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf3a8a8640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:39.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf3a8a8640 10 monclient: _renew_subs 2024-03-25T18:13:39.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf3a8a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:13:39.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf3a8a8640 1 -- 172.21.15.64:0/3099773413 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf341e7a20 con 0x7fcf3413f710 2024-03-25T18:13:39.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.118+0000 7fcf3a8a8640 1 librados: init done 2024-03-25T18:13:39.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.119+0000 7fcf3a8a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:39.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.119+0000 7fcf3a8a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:39.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.119+0000 7fcf3a8a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:13:39.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.119+0000 7fcf3a8a8640 1 -- 172.21.15.64:0/3099773413 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf340627b0 con 0x7fcf3413f710 2024-03-25T18:13:39.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.389+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf28001c90 con 0x7fcf3413f710 2024-03-25T18:13:39.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.389+0000 7fcf30ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:39.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.389+0000 7fcf327fc640 4 set_mon_vals no callback set 2024-03-25T18:13:39.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.389+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcf28003a60 con 0x7fcf3413f710 2024-03-25T18:13:39.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.389+0000 7fcf30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:39.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.389+0000 7fcf30ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:39.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.389+0000 7fcf30ff9640 10 monclient: dump: 2024-03-25T18:13:39.390 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:39.390 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:39.391 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:39.391 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:39.391 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:39.391 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:39.391 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:39.391 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:39.391 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:39.391 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:39.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.390+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fcf28003c80 con 0x7fcf3413f710 2024-03-25T18:13:39.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.391+0000 7fcf33fff640 10 monclient: get_auth_request con 0x7fcf2006de80 auth_method 0 2024-03-25T18:13:39.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.391+0000 7fcf30ff9640 10 client.4402.objecter ms_handle_connect 0x7fcf2006de80 2024-03-25T18:13:39.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.396+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fcf28054880 con 0x7fcf3413f710 2024-03-25T18:13:39.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.396+0000 7fcf30ff9640 10 client.4402.objecter ms_dispatch 0x7fcf34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:39.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.396+0000 7fcf30ff9640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:39.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.396+0000 7fcf30ff9640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:39.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.396+0000 7fcf30ff9640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-25T18:13:39.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.397+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcf28050390 con 0x7fcf3413f710 2024-03-25T18:13:39.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.397+0000 7fcf30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:39.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.398+0000 7fcf30ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:39.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.660+0000 7fcf3a8a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:39.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.660+0000 7fcf3a8a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:39.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.660+0000 7fcf3a8a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:13:39.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.660+0000 7fcf3a8a8640 1 -- 172.21.15.64:0/3099773413 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf340b7180 con 0x7fcf3413f710 2024-03-25T18:13:39.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.686+0000 7fcf30ff9640 1 -- 172.21.15.64:0/3099773413 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fcf280522c0 con 0x7fcf3413f710 2024-03-25T18:13:39.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.687+0000 7fcf30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:39.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.687+0000 7fcf30ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:39.688 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:13:39.688 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:39.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.689+0000 7fcf3a8a8640 10 librados: watch_flush enter 2024-03-25T18:13:39.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.689+0000 7fcf3a8a8640 10 librados: watch_flush exit 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.689+0000 7fcf3a8a8640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.689+0000 7fcf3a8a8640 10 client.4402.objecter successfully canceled tick 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.689+0000 7fcf3a8a8640 1 -- 172.21.15.64:0/3099773413 >> v1:172.21.15.64:6800/2915543638 conn(0x7fcf2006de80 legacy=0x7fcf20070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.689+0000 7fcf3a8a8640 10 monclient: shutdown 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.689+0000 7fcf3a8a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.689+0000 7fcf3a8a8640 1 -- 172.21.15.64:0/3099773413 >> v1:172.21.15.86:6789/0 conn(0x7fcf3413f710 legacy=0x7fcf34070930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.689+0000 7fcf3a8a8640 1 -- 172.21.15.64:0/3099773413 shutdown_connections 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.689+0000 7fcf3a8a8640 1 -- 172.21.15.64:0/3099773413 >> 172.21.15.64:0/3099773413 conn(0x7fcf340b7ea0 msgr2=0x7fcf34074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.689+0000 7fcf3a8a8640 1 -- 172.21.15.64:0/3099773413 shutdown_connections 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.689+0000 7fcf3a8a8640 1 -- 172.21.15.64:0/3099773413 wait complete. 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:39.690+0000 7fcf3a8a8640 1 librados: shutdown 2024-03-25T18:13:39.702 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:46.508 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:46.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.620+0000 7f76f940f640 10 monclient: get_monmap_and_config 2024-03-25T18:13:46.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.620+0000 7f76f940f640 10 monclient: build_initial_monmap 2024-03-25T18:13:46.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.620+0000 7f76f940f640 10 monclient: monmap: 2024-03-25T18:13:46.621 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:13:46.621 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:46.621 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:13:46.621369+0000 2024-03-25T18:13:46.621 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:13:46.621369+0000 2024-03-25T18:13:46.622 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:46.622 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:46.622 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:13:46.622 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:13:46.622 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:13:46.622 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:46.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.621+0000 7f76f940f640 1 Processor -- start 2024-03-25T18:13:46.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.621+0000 7f76f940f640 1 -- start start 2024-03-25T18:13:46.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.621+0000 7f76f940f640 10 monclient: init 2024-03-25T18:13:46.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.621+0000 7f76f940f640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:46.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.621+0000 7f76f940f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:13:46.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.621+0000 7f76f940f640 10 monclient(hunting): picked mon.noname-b con 0x7f76f4061830 addr v1:172.21.15.64:6790/0 2024-03-25T18:13:46.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.621+0000 7f76f940f640 10 monclient(hunting): picked mon.noname-a con 0x7f76f413c8c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:13:46.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.621+0000 7f76f940f640 10 monclient(hunting): picked mon.noname-c con 0x7f76f4072aa0 addr v1:172.21.15.86:6789/0 2024-03-25T18:13:46.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.621+0000 7f76f940f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f4071d40 con 0x7f76f413c8c0 2024-03-25T18:13:46.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.621+0000 7f76f940f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f4072ef0 con 0x7f76f4072aa0 2024-03-25T18:13:46.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.621+0000 7f76f940f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f4071f70 con 0x7f76f4061830 2024-03-25T18:13:46.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.621+0000 7f76f940f640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:46.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.621+0000 7f76f940f640 10 monclient(hunting): authenticate will time out at 7212.860419s 2024-03-25T18:13:46.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.636+0000 7f76f37fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f76f4072aa0 0x7f76f414a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:34316/0 (socket says 172.21.15.64:34316) 2024-03-25T18:13:46.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.636+0000 7f76f37fe640 1 -- 172.21.15.64:0/2325446265 learned_addr learned my addr 172.21.15.64:0/2325446265 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:13:46.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.650+0000 7f76f1ffb640 1 -- 172.21.15.64:0/2325446265 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155889400 0 0) 0x7f76f4072ef0 con 0x7f76f4072aa0 2024-03-25T18:13:46.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.650+0000 7f76f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:46.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.650+0000 7f76f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:46.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.650+0000 7f76f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:46.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.650+0000 7f76f1ffb640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:13:46.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.650+0000 7f76f1ffb640 1 -- 172.21.15.64:0/2325446265 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76dc003520 con 0x7f76f4072aa0 2024-03-25T18:13:46.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.652+0000 7f76f1ffb640 1 -- 172.21.15.64:0/2325446265 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708619465 0 0) 0x7f76f4071f70 con 0x7f76f4061830 2024-03-25T18:13:46.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.652+0000 7f76f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:46.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.652+0000 7f76f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:46.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.652+0000 7f76f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:46.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.652+0000 7f76f1ffb640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:13:46.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.652+0000 7f76f1ffb640 1 -- 172.21.15.64:0/2325446265 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76f4072ef0 con 0x7f76f4061830 2024-03-25T18:13:46.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.652+0000 7f76f1ffb640 1 -- 172.21.15.64:0/2325446265 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226904779 0 0) 0x7f76f4071d40 con 0x7f76f413c8c0 2024-03-25T18:13:46.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.652+0000 7f76f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:46.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.652+0000 7f76f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:46.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.652+0000 7f76f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:46.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.652+0000 7f76f1ffb640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:13:46.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.652+0000 7f76f1ffb640 1 -- 172.21.15.64:0/2325446265 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76f4071f70 con 0x7f76f413c8c0 2024-03-25T18:13:46.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.664+0000 7f76f1ffb640 1 -- 172.21.15.64:0/2325446265 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3720623213 0 0) 0x7f76dc003520 con 0x7f76f4072aa0 2024-03-25T18:13:46.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.664+0000 7f76f1ffb640 1 -- 172.21.15.64:0/2325446265 >> v1:172.21.15.64:6790/0 conn(0x7f76f4061830 legacy=0x7f76f413c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:46.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.664+0000 7f76f1ffb640 1 -- 172.21.15.64:0/2325446265 >> v1:172.21.15.64:6789/0 conn(0x7f76f413c8c0 legacy=0x7f76f4072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:46.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.664+0000 7f76f1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:46.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.664+0000 7f76f1ffb640 1 monclient: found mon.noname-c 2024-03-25T18:13:46.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.664+0000 7f76f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:46.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.664+0000 7f76f1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:13:46.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.664+0000 7f76f1ffb640 1 -- 172.21.15.64:0/2325446265 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f406fbb0 con 0x7f76f4072aa0 2024-03-25T18:13:46.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.664+0000 7f76f1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:46.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.664+0000 7f76f1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:46.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.664+0000 7f76f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:46.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.664+0000 7f76f1ffb640 1 -- 172.21.15.64:0/2325446265 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f76ec001a10 con 0x7f76f4072aa0 2024-03-25T18:13:46.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.664+0000 7f76f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:46.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.664+0000 7f76f1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:46.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.664+0000 7f76f1ffb640 10 monclient: dump: 2024-03-25T18:13:46.666 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:46.666 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:46.666 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:46.666 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:46.666 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:46.666 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:46.666 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:46.666 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:46.666 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:46.666 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:46.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.664+0000 7f76f940f640 5 monclient: authenticate success, global_id 4405 2024-03-25T18:13:46.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.664+0000 7f76f940f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:46.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.676+0000 7f76f1ffb640 1 -- 172.21.15.64:0/2325446265 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76ec001b70 con 0x7f76f4072aa0 2024-03-25T18:13:46.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.676+0000 7f76f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:46.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.676+0000 7f76f1ffb640 1 -- 172.21.15.64:0/2325446265 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f76ec001f80 con 0x7f76f4072aa0 2024-03-25T18:13:46.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.676+0000 7f76f940f640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:46.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.676+0000 7f76f940f640 4 set_mon_vals no callback set 2024-03-25T18:13:46.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.676+0000 7f76f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:46.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.676+0000 7f76f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:46.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.676+0000 7f76f1ffb640 10 monclient: dump: 2024-03-25T18:13:46.677 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:46.677 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:46.677 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:46.677 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:46.677 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:46.678 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:46.678 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:46.678 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:46.678 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:46.678 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:46.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.676+0000 7f76f940f640 10 monclient: shutdown 2024-03-25T18:13:46.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.676+0000 7f76f940f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:46.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.676+0000 7f76f940f640 1 -- 172.21.15.64:0/2325446265 >> v1:172.21.15.86:6789/0 conn(0x7f76f4072aa0 legacy=0x7f76f414a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:46.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.677+0000 7f76f940f640 1 -- 172.21.15.64:0/2325446265 shutdown_connections 2024-03-25T18:13:46.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.677+0000 7f76f940f640 1 -- 172.21.15.64:0/2325446265 >> 172.21.15.64:0/2325446265 conn(0x7f76f40b0ef0 msgr2=0x7f76f40b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:46.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.677+0000 7f76f940f640 1 -- 172.21.15.64:0/2325446265 shutdown_connections 2024-03-25T18:13:46.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.677+0000 7f76f940f640 1 -- 172.21.15.64:0/2325446265 wait complete. 2024-03-25T18:13:46.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.677+0000 7f76f940f640 10 monclient: build_initial_monmap 2024-03-25T18:13:46.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.677+0000 7f76f940f640 10 monclient: monmap: 2024-03-25T18:13:46.681 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:13:46.681 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:46.681 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:13:46.681 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:13:46.681 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:46.681 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:13:46.681 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:13:46.681 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:13:46.681 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:13:46.681 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:46.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.678+0000 7f76f940f640 1 librados: starting msgr at 2024-03-25T18:13:46.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.678+0000 7f76f940f640 1 librados: starting objecter 2024-03-25T18:13:46.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.678+0000 7f76f940f640 1 Processor -- start 2024-03-25T18:13:46.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.678+0000 7f76f940f640 1 -- start start 2024-03-25T18:13:46.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.678+0000 7f76f940f640 1 librados: setting wanted keys 2024-03-25T18:13:46.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.678+0000 7f76f940f640 1 librados: calling monclient init 2024-03-25T18:13:46.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.678+0000 7f76f940f640 10 monclient: init 2024-03-25T18:13:46.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.679+0000 7f76f940f640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:46.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.679+0000 7f76f940f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:46.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.679+0000 7f76f940f640 10 monclient(hunting): picked mon.noname-c con 0x7f76f4061830 addr v1:172.21.15.64:6790/0 2024-03-25T18:13:46.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.679+0000 7f76f940f640 10 monclient(hunting): picked mon.noname-b con 0x7f76f4072aa0 addr v1:172.21.15.86:6789/0 2024-03-25T18:13:46.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.679+0000 7f76f940f640 10 monclient(hunting): picked mon.noname-a con 0x7f76f413c8c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:13:46.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.679+0000 7f76f940f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f413f620 con 0x7f76f413c8c0 2024-03-25T18:13:46.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.679+0000 7f76f940f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f41dafd0 con 0x7f76f4072aa0 2024-03-25T18:13:46.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.679+0000 7f76f940f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f41360d0 con 0x7f76f4061830 2024-03-25T18:13:46.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.679+0000 7f76f940f640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:46.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.679+0000 7f76f940f640 10 monclient(hunting): authenticate will time out at 7212.917842s 2024-03-25T18:13:46.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.685+0000 7f76f2ffd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f76f4061830 0x7f76f413cdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:60142/0 (socket says 172.21.15.64:60142) 2024-03-25T18:13:46.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.685+0000 7f76f2ffd640 1 -- 172.21.15.64:0/1275367011 learned_addr learned my addr 172.21.15.64:0/1275367011 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:13:46.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.690+0000 7f76db7fe640 10 client.?.objecter ms_handle_connect 0x7f76f4061830 2024-03-25T18:13:46.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.690+0000 7f76db7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:46.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.698+0000 7f76db7fe640 10 client.?.objecter ms_handle_connect 0x7f76f413c8c0 2024-03-25T18:13:46.699 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.698+0000 7f76db7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:46.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.699+0000 7f76db7fe640 10 client.?.objecter ms_handle_connect 0x7f76f4072aa0 2024-03-25T18:13:46.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.699+0000 7f76db7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:46.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.707+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715415592 0 0) 0x7f76f413f620 con 0x7f76f413c8c0 2024-03-25T18:13:46.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.707+0000 7f76db7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:46.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.707+0000 7f76db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:46.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.707+0000 7f76db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:46.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.707+0000 7f76db7fe640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:13:46.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.707+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e0004eb0 con 0x7f76f413c8c0 2024-03-25T18:13:46.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.707+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769311032 0 0) 0x7f76f41dafd0 con 0x7f76f4072aa0 2024-03-25T18:13:46.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.708+0000 7f76db7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:46.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.708+0000 7f76db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:46.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.708+0000 7f76db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:46.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.708+0000 7f76db7fe640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:13:46.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.708+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76f413f620 con 0x7f76f4072aa0 2024-03-25T18:13:46.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.711+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830706852 0 0) 0x7f76f41360d0 con 0x7f76f4061830 2024-03-25T18:13:46.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.711+0000 7f76db7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:46.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.711+0000 7f76db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:46.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.711+0000 7f76db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:46.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.711+0000 7f76db7fe640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:13:46.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.711+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76f41dafd0 con 0x7f76f4061830 2024-03-25T18:13:46.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.724+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3395107709 0 0) 0x7f76f413f620 con 0x7f76f4072aa0 2024-03-25T18:13:46.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.724+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76f41360d0 con 0x7f76f4072aa0 2024-03-25T18:13:46.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.724+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f76e8001810 con 0x7f76f4072aa0 2024-03-25T18:13:46.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.724+0000 7f76db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:46.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.724+0000 7f76db7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:46.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.724+0000 7f76db7fe640 10 monclient(hunting): dump: 2024-03-25T18:13:46.725 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:46.726 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:46.726 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:46.726 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:46.726 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:46.726 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:46.726 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:46.726 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:46.726 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:46.726 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:46.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.724+0000 7f76db7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:46.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.725+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2689307579 0 0) 0x7f76e0004eb0 con 0x7f76f413c8c0 2024-03-25T18:13:46.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.725+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76f413f620 con 0x7f76f413c8c0 2024-03-25T18:13:46.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.725+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f76ec001e80 con 0x7f76f413c8c0 2024-03-25T18:13:46.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.725+0000 7f76db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:46.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.725+0000 7f76db7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:46.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.725+0000 7f76db7fe640 10 monclient(hunting): dump: 2024-03-25T18:13:46.727 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:46.727 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:46.727 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:46.727 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:46.727 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:46.727 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:46.727 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:46.728 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:46.728 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:46.728 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:46.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.728+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 268117386 0 0) 0x7f76f41dafd0 con 0x7f76f4061830 2024-03-25T18:13:46.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.728+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76e0004eb0 con 0x7f76f4061830 2024-03-25T18:13:46.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.728+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f76e4001930 con 0x7f76f4061830 2024-03-25T18:13:46.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.728+0000 7f76db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:46.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.728+0000 7f76db7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:46.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.728+0000 7f76db7fe640 10 monclient(hunting): dump: 2024-03-25T18:13:46.729 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:46.729 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:46.729 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:46.730 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:46.730 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:46.730 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:46.730 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:46.730 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:46.730 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:46.730 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:46.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2519539562 0 0) 0x7f76e0004eb0 con 0x7f76f4061830 2024-03-25T18:13:46.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 >> v1:172.21.15.86:6789/0 conn(0x7f76f4072aa0 legacy=0x7f76f41da8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:46.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 >> v1:172.21.15.64:6789/0 conn(0x7f76f413c8c0 legacy=0x7f76f41359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:46.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76db7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:46.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76db7fe640 1 monclient: found mon.c 2024-03-25T18:13:46.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:46.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76db7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:13:46.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f4136260 con 0x7f76f4061830 2024-03-25T18:13:46.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76db7fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:46.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76db7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:46.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:46.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76f940f640 5 monclient: authenticate success, global_id 4415 2024-03-25T18:13:46.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76f940f640 10 monclient: _renew_subs 2024-03-25T18:13:46.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76f940f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:13:46.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76f940f640 1 -- 172.21.15.64:0/1275367011 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f413fb20 con 0x7f76f4061830 2024-03-25T18:13:46.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76f940f640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:46.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76f940f640 10 monclient: _renew_subs 2024-03-25T18:13:46.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76f940f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:13:46.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76f940f640 1 -- 172.21.15.64:0/1275367011 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f413dbd0 con 0x7f76f4061830 2024-03-25T18:13:46.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.737+0000 7f76f940f640 1 librados: init done 2024-03-25T18:13:46.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.739+0000 7f76f940f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:46.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.739+0000 7f76f940f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:46.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.739+0000 7f76f940f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:13:46.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:46.739+0000 7f76f940f640 1 -- 172.21.15.64:0/1275367011 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f413f440 con 0x7f76f4061830 2024-03-25T18:13:47.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.106+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76e4003610 con 0x7f76f4061830 2024-03-25T18:13:47.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.106+0000 7f76db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:47.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.106+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f76e4003960 con 0x7f76f4061830 2024-03-25T18:13:47.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.106+0000 7f76db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:47.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.106+0000 7f76db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:47.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.106+0000 7f76db7fe640 10 monclient: dump: 2024-03-25T18:13:47.107 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:47.107 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:47.107 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:47.107 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:47.107 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:47.108 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:47.108 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:47.108 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:47.108 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:47.108 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:47.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.106+0000 7f76f17fa640 4 set_mon_vals no callback set 2024-03-25T18:13:47.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.106+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f76e4003b80 con 0x7f76f4061830 2024-03-25T18:13:47.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.107+0000 7f76f27fc640 10 monclient: get_auth_request con 0x7f76e006e160 auth_method 0 2024-03-25T18:13:47.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.107+0000 7f76db7fe640 10 client.4415.objecter ms_handle_connect 0x7f76e006e160 2024-03-25T18:13:47.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.113+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f76e40547a0 con 0x7f76f4061830 2024-03-25T18:13:47.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.113+0000 7f76db7fe640 10 client.4415.objecter ms_dispatch 0x7f76f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:47.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.113+0000 7f76db7fe640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:47.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.113+0000 7f76db7fe640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:47.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.113+0000 7f76db7fe640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-25T18:13:47.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.113+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76e40502b0 con 0x7f76f4061830 2024-03-25T18:13:47.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.113+0000 7f76db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:47.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.113+0000 7f76db7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:47.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.350+0000 7f76f940f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:47.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.350+0000 7f76f940f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:47.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.350+0000 7f76f940f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:13:47.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.350+0000 7f76f940f640 1 -- 172.21.15.64:0/1275367011 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76f40627b0 con 0x7f76f4061830 2024-03-25T18:13:47.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.377+0000 7f76db7fe640 1 -- 172.21.15.64:0/1275367011 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f76e40521e0 con 0x7f76f4061830 2024-03-25T18:13:47.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.377+0000 7f76db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:47.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.377+0000 7f76db7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:47.378 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:13:47.379 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:47.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.379+0000 7f76f940f640 10 librados: watch_flush enter 2024-03-25T18:13:47.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.379+0000 7f76f940f640 10 librados: watch_flush exit 2024-03-25T18:13:47.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.379+0000 7f76f940f640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-25T18:13:47.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.379+0000 7f76f940f640 10 client.4415.objecter successfully canceled tick 2024-03-25T18:13:47.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.379+0000 7f76f940f640 1 -- 172.21.15.64:0/1275367011 >> v1:172.21.15.64:6800/2915543638 conn(0x7f76e006e160 legacy=0x7f76e0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:47.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.379+0000 7f76f940f640 10 monclient: shutdown 2024-03-25T18:13:47.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.379+0000 7f76f940f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:47.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.379+0000 7f76f940f640 1 -- 172.21.15.64:0/1275367011 >> v1:172.21.15.64:6790/0 conn(0x7f76f4061830 legacy=0x7f76f413cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:47.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.380+0000 7f76f940f640 1 -- 172.21.15.64:0/1275367011 shutdown_connections 2024-03-25T18:13:47.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.380+0000 7f76f940f640 1 -- 172.21.15.64:0/1275367011 >> 172.21.15.64:0/1275367011 conn(0x7f76f40b0ef0 msgr2=0x7f76f413bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:47.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.380+0000 7f76f940f640 1 -- 172.21.15.64:0/1275367011 shutdown_connections 2024-03-25T18:13:47.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.380+0000 7f76f940f640 1 -- 172.21.15.64:0/1275367011 wait complete. 2024-03-25T18:13:47.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:47.380+0000 7f76f940f640 1 librados: shutdown 2024-03-25T18:13:47.392 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:54.198 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:54.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.309+0000 7fe069f9d640 10 monclient: get_monmap_and_config 2024-03-25T18:13:54.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.309+0000 7fe069f9d640 10 monclient: build_initial_monmap 2024-03-25T18:13:54.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.309+0000 7fe069f9d640 10 monclient: monmap: 2024-03-25T18:13:54.310 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:13:54.310 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:54.310 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:13:54.310578+0000 2024-03-25T18:13:54.311 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:13:54.310578+0000 2024-03-25T18:13:54.311 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:54.311 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:54.311 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:13:54.311 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:13:54.311 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:13:54.311 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:54.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.310+0000 7fe069f9d640 1 Processor -- start 2024-03-25T18:13:54.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.310+0000 7fe069f9d640 1 -- start start 2024-03-25T18:13:54.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.310+0000 7fe069f9d640 10 monclient: init 2024-03-25T18:13:54.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.311+0000 7fe069f9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:54.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.311+0000 7fe069f9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:54.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.311+0000 7fe069f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe0640b4f40 addr v1:172.21.15.64:6789/0 2024-03-25T18:13:54.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.311+0000 7fe069f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe06413fd90 addr v1:172.21.15.64:6790/0 2024-03-25T18:13:54.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.311+0000 7fe069f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe0640b4540 addr v1:172.21.15.86:6789/0 2024-03-25T18:13:54.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.311+0000 7fe069f9d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0640aa0c0 con 0x7fe0640b4f40 2024-03-25T18:13:54.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.311+0000 7fe069f9d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0640ab170 con 0x7fe0640b4540 2024-03-25T18:13:54.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.311+0000 7fe069f9d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0640a9ba0 con 0x7fe06413fd90 2024-03-25T18:13:54.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.311+0000 7fe069f9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:54.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.311+0000 7fe069f9d640 10 monclient(hunting): authenticate will time out at 7220.550264s 2024-03-25T18:13:54.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.324+0000 7fe063fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe0640b4540 0x7fe06414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:38970/0 (socket says 172.21.15.64:38970) 2024-03-25T18:13:54.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.324+0000 7fe063fff640 1 -- 172.21.15.64:0/559769701 learned_addr learned my addr 172.21.15.64:0/559769701 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:13:54.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.335+0000 7fe0627fc640 1 -- 172.21.15.64:0/559769701 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956078973 0 0) 0x7fe0640ab170 con 0x7fe0640b4540 2024-03-25T18:13:54.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.335+0000 7fe0627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:54.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.335+0000 7fe0627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:54.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.335+0000 7fe0627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:54.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.335+0000 7fe0627fc640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:13:54.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.335+0000 7fe0627fc640 1 -- 172.21.15.64:0/559769701 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe04c003520 con 0x7fe0640b4540 2024-03-25T18:13:54.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.342+0000 7fe0627fc640 1 -- 172.21.15.64:0/559769701 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957066988 0 0) 0x7fe0640a9ba0 con 0x7fe06413fd90 2024-03-25T18:13:54.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.342+0000 7fe0627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:54.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.342+0000 7fe0627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:54.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.342+0000 7fe0627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:54.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.342+0000 7fe0627fc640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:13:54.344 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.342+0000 7fe0627fc640 1 -- 172.21.15.64:0/559769701 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0640ab170 con 0x7fe06413fd90 2024-03-25T18:13:54.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.345+0000 7fe0627fc640 1 -- 172.21.15.64:0/559769701 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2355264867 0 0) 0x7fe04c003520 con 0x7fe0640b4540 2024-03-25T18:13:54.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.345+0000 7fe0627fc640 1 -- 172.21.15.64:0/559769701 >> v1:172.21.15.64:6790/0 conn(0x7fe06413fd90 legacy=0x7fe0641423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.345+0000 7fe0627fc640 1 -- 172.21.15.64:0/559769701 >> v1:172.21.15.64:6789/0 conn(0x7fe0640b4f40 legacy=0x7fe06413f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.345+0000 7fe0627fc640 10 monclient: _finish_hunting 0 2024-03-25T18:13:54.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.345+0000 7fe0627fc640 1 monclient: found mon.noname-c 2024-03-25T18:13:54.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.345+0000 7fe0627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:54.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.346+0000 7fe0627fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:13:54.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.346+0000 7fe0627fc640 1 -- 172.21.15.64:0/559769701 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0640a9680 con 0x7fe0640b4540 2024-03-25T18:13:54.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.346+0000 7fe0627fc640 10 monclient: _finish_auth 0 2024-03-25T18:13:54.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.346+0000 7fe0627fc640 10 monclient: _check_auth_tickets 2024-03-25T18:13:54.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.346+0000 7fe0627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:54.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.346+0000 7fe0627fc640 1 -- 172.21.15.64:0/559769701 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe05c001a10 con 0x7fe0640b4540 2024-03-25T18:13:54.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.346+0000 7fe069f9d640 5 monclient: authenticate success, global_id 4411 2024-03-25T18:13:54.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.346+0000 7fe069f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:54.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.346+0000 7fe0627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:54.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.346+0000 7fe0627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:54.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.346+0000 7fe0627fc640 10 monclient: dump: 2024-03-25T18:13:54.348 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:54.348 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:54.348 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:54.348 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:54.348 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:54.348 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:54.349 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:54.349 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:54.349 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:54.349 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:54.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.346+0000 7fe069f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:54.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.363+0000 7fe0627fc640 1 -- 172.21.15.64:0/559769701 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe05c001b70 con 0x7fe0640b4540 2024-03-25T18:13:54.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.363+0000 7fe0627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:54.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.363+0000 7fe0627fc640 1 -- 172.21.15.64:0/559769701 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe05c001f80 con 0x7fe0640b4540 2024-03-25T18:13:54.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.363+0000 7fe069f9d640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:54.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.363+0000 7fe069f9d640 4 set_mon_vals no callback set 2024-03-25T18:13:54.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.363+0000 7fe0627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:54.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.363+0000 7fe0627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:54.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.363+0000 7fe0627fc640 10 monclient: dump: 2024-03-25T18:13:54.365 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:54.365 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:54.365 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:54.365 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:54.365 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:54.365 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:54.365 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:54.365 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:54.365 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:54.365 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:54.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.364+0000 7fe069f9d640 10 monclient: shutdown 2024-03-25T18:13:54.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.364+0000 7fe069f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:54.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.364+0000 7fe069f9d640 1 -- 172.21.15.64:0/559769701 >> v1:172.21.15.86:6789/0 conn(0x7fe0640b4540 legacy=0x7fe06414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.364+0000 7fe069f9d640 1 -- 172.21.15.64:0/559769701 shutdown_connections 2024-03-25T18:13:54.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.364+0000 7fe069f9d640 1 -- 172.21.15.64:0/559769701 >> 172.21.15.64:0/559769701 conn(0x7fe0640617a0 msgr2=0x7fe064061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:54.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.364+0000 7fe069f9d640 1 -- 172.21.15.64:0/559769701 shutdown_connections 2024-03-25T18:13:54.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.364+0000 7fe069f9d640 1 -- 172.21.15.64:0/559769701 wait complete. 2024-03-25T18:13:54.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.365+0000 7fe069f9d640 10 monclient: build_initial_monmap 2024-03-25T18:13:54.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.365+0000 7fe069f9d640 10 monclient: monmap: 2024-03-25T18:13:54.367 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:13:54.367 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:54.367 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:13:54.368 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:13:54.368 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:54.368 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:13:54.368 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:13:54.368 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:13:54.368 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:13:54.368 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:54.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.365+0000 7fe069f9d640 1 librados: starting msgr at 2024-03-25T18:13:54.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.365+0000 7fe069f9d640 1 librados: starting objecter 2024-03-25T18:13:54.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.365+0000 7fe069f9d640 1 Processor -- start 2024-03-25T18:13:54.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.366+0000 7fe069f9d640 1 -- start start 2024-03-25T18:13:54.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.366+0000 7fe069f9d640 1 librados: setting wanted keys 2024-03-25T18:13:54.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.366+0000 7fe069f9d640 1 librados: calling monclient init 2024-03-25T18:13:54.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.366+0000 7fe069f9d640 10 monclient: init 2024-03-25T18:13:54.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.366+0000 7fe069f9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:54.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.366+0000 7fe069f9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:54.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.366+0000 7fe069f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fe0640b4540 addr v1:172.21.15.64:6789/0 2024-03-25T18:13:54.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.367+0000 7fe069f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fe0640b4f40 addr v1:172.21.15.64:6790/0 2024-03-25T18:13:54.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.367+0000 7fe069f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe06413fd90 addr v1:172.21.15.86:6789/0 2024-03-25T18:13:54.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.367+0000 7fe069f9d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0640afc20 con 0x7fe0640b4540 2024-03-25T18:13:54.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.367+0000 7fe069f9d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0640acfd0 con 0x7fe06413fd90 2024-03-25T18:13:54.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.367+0000 7fe069f9d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0640b0e80 con 0x7fe0640b4f40 2024-03-25T18:13:54.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.367+0000 7fe069f9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:54.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.367+0000 7fe069f9d640 10 monclient(hunting): authenticate will time out at 7220.605641s 2024-03-25T18:13:54.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.374+0000 7fe063fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe06413fd90 0x7fe0641382f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:39022/0 (socket says 172.21.15.64:39022) 2024-03-25T18:13:54.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.374+0000 7fe063fff640 1 -- 172.21.15.64:0/2125762195 learned_addr learned my addr 172.21.15.64:0/2125762195 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:13:54.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.376+0000 7fe04bfff640 10 client.?.objecter ms_handle_connect 0x7fe06413fd90 2024-03-25T18:13:54.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.376+0000 7fe04bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:54.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.377+0000 7fe04bfff640 10 client.?.objecter ms_handle_connect 0x7fe0640b4540 2024-03-25T18:13:54.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.377+0000 7fe04bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:54.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.381+0000 7fe04bfff640 10 client.?.objecter ms_handle_connect 0x7fe0640b4f40 2024-03-25T18:13:54.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.381+0000 7fe04bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:54.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.382+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043693294 0 0) 0x7fe0640afc20 con 0x7fe0640b4540 2024-03-25T18:13:54.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.382+0000 7fe04bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:54.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.382+0000 7fe04bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:54.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.382+0000 7fe04bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:54.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.382+0000 7fe04bfff640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:13:54.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.382+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe050004eb0 con 0x7fe0640b4540 2024-03-25T18:13:54.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.386+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363878329 0 0) 0x7fe0640acfd0 con 0x7fe06413fd90 2024-03-25T18:13:54.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.386+0000 7fe04bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:54.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.386+0000 7fe04bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:54.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.386+0000 7fe04bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:54.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.386+0000 7fe04bfff640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:13:54.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.386+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0640afc20 con 0x7fe06413fd90 2024-03-25T18:13:54.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.394+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3428169381 0 0) 0x7fe0640afc20 con 0x7fe06413fd90 2024-03-25T18:13:54.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.394+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0640acfd0 con 0x7fe06413fd90 2024-03-25T18:13:54.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.394+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe05c001e80 con 0x7fe06413fd90 2024-03-25T18:13:54.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.394+0000 7fe04bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:54.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.394+0000 7fe04bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:54.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.394+0000 7fe04bfff640 10 monclient(hunting): dump: 2024-03-25T18:13:54.395 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:54.395 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:54.396 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:54.396 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:54.396 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:54.396 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:54.396 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:54.396 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:54.396 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:54.396 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:54.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.394+0000 7fe04bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:54.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.400+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212908213 0 0) 0x7fe0640b0e80 con 0x7fe0640b4f40 2024-03-25T18:13:54.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.400+0000 7fe04bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:54.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.400+0000 7fe04bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:54.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.400+0000 7fe04bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:54.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.400+0000 7fe04bfff640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:13:54.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.400+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0640afc20 con 0x7fe0640b4f40 2024-03-25T18:13:54.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.402+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3968683534 0 0) 0x7fe050004eb0 con 0x7fe0640b4540 2024-03-25T18:13:54.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.402+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0640b0e80 con 0x7fe0640b4540 2024-03-25T18:13:54.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.402+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe0540018c0 con 0x7fe0640b4540 2024-03-25T18:13:54.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.402+0000 7fe04bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:54.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.402+0000 7fe04bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:54.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.402+0000 7fe04bfff640 10 monclient(hunting): dump: 2024-03-25T18:13:54.403 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:54.403 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:54.403 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:54.404 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:54.404 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:54.404 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:54.404 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:54.404 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:54.404 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:54.404 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:54.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2523475035 0 0) 0x7fe0640acfd0 con 0x7fe06413fd90 2024-03-25T18:13:54.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 >> v1:172.21.15.64:6790/0 conn(0x7fe0640b4f40 legacy=0x7fe0640ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 >> v1:172.21.15.64:6789/0 conn(0x7fe0640b4540 legacy=0x7fe0640af250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe04bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:54.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe04bfff640 1 monclient: found mon.b 2024-03-25T18:13:54.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe04bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:54.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe04bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:13:54.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0640b10f0 con 0x7fe06413fd90 2024-03-25T18:13:54.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe04bfff640 10 monclient: _finish_auth 0 2024-03-25T18:13:54.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe04bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:54.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe04bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:54.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe069f9d640 5 monclient: authenticate success, global_id 4414 2024-03-25T18:13:54.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe069f9d640 10 monclient: _renew_subs 2024-03-25T18:13:54.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe069f9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:13:54.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe069f9d640 1 -- 172.21.15.64:0/2125762195 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0640b13d0 con 0x7fe06413fd90 2024-03-25T18:13:54.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe069f9d640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:54.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe069f9d640 10 monclient: _renew_subs 2024-03-25T18:13:54.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe069f9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:13:54.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe069f9d640 1 -- 172.21.15.64:0/2125762195 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe064138a00 con 0x7fe06413fd90 2024-03-25T18:13:54.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.415+0000 7fe069f9d640 1 librados: init done 2024-03-25T18:13:54.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.417+0000 7fe069f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:54.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.417+0000 7fe069f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:54.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.417+0000 7fe069f9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:13:54.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.417+0000 7fe069f9d640 1 -- 172.21.15.64:0/2125762195 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0640627b0 con 0x7fe06413fd90 2024-03-25T18:13:54.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.686+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe05c004e10 con 0x7fe06413fd90 2024-03-25T18:13:54.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.686+0000 7fe04bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:54.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.686+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe05c005120 con 0x7fe06413fd90 2024-03-25T18:13:54.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.686+0000 7fe04bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:54.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.686+0000 7fe04bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:54.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.686+0000 7fe04bfff640 10 monclient: dump: 2024-03-25T18:13:54.687 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:13:54.687 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:13:54.687 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:54.687 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:13:54.687 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:54.687 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:13:54.687 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:13:54.688 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:13:54.688 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:13:54.688 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:13:54.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.686+0000 7fe061ffb640 4 set_mon_vals no callback set 2024-03-25T18:13:54.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.686+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fe05c01b0f0 con 0x7fe06413fd90 2024-03-25T18:13:54.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.687+0000 7fe0637fe640 10 monclient: get_auth_request con 0x7fe05006e060 auth_method 0 2024-03-25T18:13:54.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.687+0000 7fe04bfff640 10 client.4414.objecter ms_handle_connect 0x7fe05006e060 2024-03-25T18:13:54.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.692+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fe05c054bf0 con 0x7fe06413fd90 2024-03-25T18:13:54.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.692+0000 7fe04bfff640 10 client.4414.objecter ms_dispatch 0x7fe064000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:54.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.692+0000 7fe04bfff640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:54.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.692+0000 7fe04bfff640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:54.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.692+0000 7fe04bfff640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-25T18:13:54.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.693+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe05c050700 con 0x7fe06413fd90 2024-03-25T18:13:54.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.693+0000 7fe04bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:54.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.693+0000 7fe04bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:54.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.953+0000 7fe069f9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:54.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.953+0000 7fe069f9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:54.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.953+0000 7fe069f9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:13:54.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:54.953+0000 7fe069f9d640 1 -- 172.21.15.64:0/2125762195 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe06413a4f0 con 0x7fe06413fd90 2024-03-25T18:13:55.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:55.044+0000 7fe04bfff640 1 -- 172.21.15.64:0/2125762195 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fe05c052630 con 0x7fe06413fd90 2024-03-25T18:13:55.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:55.044+0000 7fe04bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:55.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:55.044+0000 7fe04bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:55.045 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:13:55.046 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:55.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:55.047+0000 7fe069f9d640 10 librados: watch_flush enter 2024-03-25T18:13:55.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:55.047+0000 7fe069f9d640 10 librados: watch_flush exit 2024-03-25T18:13:55.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:55.047+0000 7fe069f9d640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-25T18:13:55.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:55.047+0000 7fe069f9d640 10 client.4414.objecter successfully canceled tick 2024-03-25T18:13:55.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:55.047+0000 7fe069f9d640 1 -- 172.21.15.64:0/2125762195 >> v1:172.21.15.64:6800/2915543638 conn(0x7fe05006e060 legacy=0x7fe050070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:55.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:55.047+0000 7fe069f9d640 10 monclient: shutdown 2024-03-25T18:13:55.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:55.047+0000 7fe069f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:55.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:55.047+0000 7fe069f9d640 1 -- 172.21.15.64:0/2125762195 >> v1:172.21.15.86:6789/0 conn(0x7fe06413fd90 legacy=0x7fe0641382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:55.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:55.047+0000 7fe069f9d640 1 -- 172.21.15.64:0/2125762195 shutdown_connections 2024-03-25T18:13:55.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:55.047+0000 7fe069f9d640 1 -- 172.21.15.64:0/2125762195 >> 172.21.15.64:0/2125762195 conn(0x7fe0640617a0 msgr2=0x7fe064139d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:55.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:55.047+0000 7fe069f9d640 1 -- 172.21.15.64:0/2125762195 shutdown_connections 2024-03-25T18:13:55.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:55.047+0000 7fe069f9d640 1 -- 172.21.15.64:0/2125762195 wait complete. 2024-03-25T18:13:55.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:13:55.047+0000 7fe069f9d640 1 librados: shutdown 2024-03-25T18:13:55.060 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:57.194 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:04:40.307 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:57.199 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:04:40.312 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:57.240 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:04:40.353 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:57.243 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:04:40.322 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:57.256 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:04:40.335 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:57.258 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:04:40.371 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:57.267 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:04:40.380 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:57.295 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:04:40.374 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:57.301 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:04:40.414 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:01.867 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:01.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:01.978+0000 7f99881cb640 10 monclient: get_monmap_and_config 2024-03-25T18:14:01.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:01.978+0000 7f99881cb640 10 monclient: build_initial_monmap 2024-03-25T18:14:01.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:01.978+0000 7f99881cb640 10 monclient: monmap: 2024-03-25T18:14:01.980 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:14:01.980 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:01.980 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:14:01.979759+0000 2024-03-25T18:14:01.980 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:14:01.979759+0000 2024-03-25T18:14:01.980 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:01.980 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:01.980 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:14:01.980 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:14:01.980 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:14:01.980 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:01.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:01.979+0000 7f99881cb640 1 Processor -- start 2024-03-25T18:14:01.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:01.979+0000 7f99881cb640 1 -- start start 2024-03-25T18:14:01.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:01.979+0000 7f99881cb640 10 monclient: init 2024-03-25T18:14:01.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:01.980+0000 7f99881cb640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:01.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:01.980+0000 7f99881cb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:01.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:01.980+0000 7f99881cb640 10 monclient(hunting): picked mon.noname-b con 0x7f99800a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:14:01.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:01.980+0000 7f99881cb640 10 monclient(hunting): picked mon.noname-a con 0x7f998013f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:14:01.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:01.980+0000 7f99881cb640 10 monclient(hunting): picked mon.noname-c con 0x7f99801430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:14:01.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:01.980+0000 7f99881cb640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9980142520 con 0x7f998013f710 2024-03-25T18:14:01.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:01.980+0000 7f99881cb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99800b28a0 con 0x7f99801430c0 2024-03-25T18:14:01.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:01.980+0000 7f99881cb640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99800b2a30 con 0x7f99800a98f0 2024-03-25T18:14:01.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:01.980+0000 7f99881cb640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:01.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:01.980+0000 7f99881cb640 10 monclient(hunting): authenticate will time out at 7228.218877s 2024-03-25T18:14:01.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:01.990+0000 7f9985f67640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f99800a98f0 0x7f99800a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:59956/0 (socket says 172.21.15.64:59956) 2024-03-25T18:14:01.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:01.990+0000 7f9985f67640 1 -- 172.21.15.64:0/1527619644 learned_addr learned my addr 172.21.15.64:0/1527619644 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:14:02.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.005+0000 7f9984f65640 1 -- 172.21.15.64:0/1527619644 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254175606 0 0) 0x7f99800b2a30 con 0x7f99800a98f0 2024-03-25T18:14:02.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.005+0000 7f9984f65640 10 monclient(hunting): _negotiate 2024-03-25T18:14:02.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.005+0000 7f9984f65640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:02.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.005+0000 7f9984f65640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:02.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.005+0000 7f9984f65640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:14:02.006 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.005+0000 7f9984f65640 1 -- 172.21.15.64:0/1527619644 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f996c003520 con 0x7f99800a98f0 2024-03-25T18:14:02.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.009+0000 7f9984f65640 1 -- 172.21.15.64:0/1527619644 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440071507 0 0) 0x7f99800b28a0 con 0x7f99801430c0 2024-03-25T18:14:02.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.009+0000 7f9984f65640 10 monclient(hunting): _negotiate 2024-03-25T18:14:02.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.009+0000 7f9984f65640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:02.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.009+0000 7f9984f65640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:02.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.009+0000 7f9984f65640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:14:02.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.010+0000 7f9984f65640 1 -- 172.21.15.64:0/1527619644 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99800b2a30 con 0x7f99801430c0 2024-03-25T18:14:02.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.021+0000 7f9984f65640 1 -- 172.21.15.64:0/1527619644 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764994854 0 0) 0x7f9980142520 con 0x7f998013f710 2024-03-25T18:14:02.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.021+0000 7f9984f65640 10 monclient(hunting): _negotiate 2024-03-25T18:14:02.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.021+0000 7f9984f65640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:02.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.021+0000 7f9984f65640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:02.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.021+0000 7f9984f65640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:14:02.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.021+0000 7f9984f65640 1 -- 172.21.15.64:0/1527619644 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99800b28a0 con 0x7f998013f710 2024-03-25T18:14:02.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.031+0000 7f9984f65640 1 -- 172.21.15.64:0/1527619644 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3036148582 0 0) 0x7f996c003520 con 0x7f99800a98f0 2024-03-25T18:14:02.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.031+0000 7f9984f65640 1 -- 172.21.15.64:0/1527619644 >> v1:172.21.15.86:6789/0 conn(0x7f99801430c0 legacy=0x7f998014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:02.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.031+0000 7f9984f65640 1 -- 172.21.15.64:0/1527619644 >> v1:172.21.15.64:6789/0 conn(0x7f998013f710 legacy=0x7f99801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:02.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.031+0000 7f9984f65640 10 monclient: _finish_hunting 0 2024-03-25T18:14:02.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.031+0000 7f9984f65640 1 monclient: found mon.noname-b 2024-03-25T18:14:02.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.031+0000 7f9984f65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:02.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.031+0000 7f9984f65640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:14:02.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.031+0000 7f9984f65640 1 -- 172.21.15.64:0/1527619644 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99800b44c0 con 0x7f99800a98f0 2024-03-25T18:14:02.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.031+0000 7f9984f65640 10 monclient: _finish_auth 0 2024-03-25T18:14:02.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.031+0000 7f9984f65640 10 monclient: _check_auth_tickets 2024-03-25T18:14:02.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.031+0000 7f9984f65640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:02.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.031+0000 7f9984f65640 1 -- 172.21.15.64:0/1527619644 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f997c001690 con 0x7f99800a98f0 2024-03-25T18:14:02.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.031+0000 7f9984f65640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:02.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.031+0000 7f9984f65640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:02.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.031+0000 7f9984f65640 10 monclient: dump: 2024-03-25T18:14:02.034 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:02.034 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:02.034 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:02.034 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:02.034 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:02.034 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:02.034 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:02.034 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:02.034 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:02.034 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:02.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.032+0000 7f99881cb640 5 monclient: authenticate success, global_id 4424 2024-03-25T18:14:02.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.032+0000 7f99881cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:02.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.042+0000 7f9984f65640 1 -- 172.21.15.64:0/1527619644 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f997c0017f0 con 0x7f99800a98f0 2024-03-25T18:14:02.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.042+0000 7f9984f65640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:02.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.042+0000 7f9984f65640 1 -- 172.21.15.64:0/1527619644 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f997c001c00 con 0x7f99800a98f0 2024-03-25T18:14:02.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.042+0000 7f99881cb640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:02.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.042+0000 7f99881cb640 4 set_mon_vals no callback set 2024-03-25T18:14:02.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.042+0000 7f9984f65640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:02.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.042+0000 7f9984f65640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:02.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.042+0000 7f9984f65640 10 monclient: dump: 2024-03-25T18:14:02.044 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:02.044 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:02.044 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:02.044 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:02.044 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:02.044 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:02.044 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:02.044 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:02.044 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:02.044 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:02.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.043+0000 7f99881cb640 10 monclient: shutdown 2024-03-25T18:14:02.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.043+0000 7f99881cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:02.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.043+0000 7f99881cb640 1 -- 172.21.15.64:0/1527619644 >> v1:172.21.15.64:6790/0 conn(0x7f99800a98f0 legacy=0x7f99800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:02.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.043+0000 7f99881cb640 1 -- 172.21.15.64:0/1527619644 shutdown_connections 2024-03-25T18:14:02.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.043+0000 7f99881cb640 1 -- 172.21.15.64:0/1527619644 >> 172.21.15.64:0/1527619644 conn(0x7f99800b7ea0 msgr2=0x7f99800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:02.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.043+0000 7f99881cb640 1 -- 172.21.15.64:0/1527619644 shutdown_connections 2024-03-25T18:14:02.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.044+0000 7f99881cb640 1 -- 172.21.15.64:0/1527619644 wait complete. 2024-03-25T18:14:02.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.044+0000 7f99881cb640 10 monclient: build_initial_monmap 2024-03-25T18:14:02.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.044+0000 7f99881cb640 10 monclient: monmap: 2024-03-25T18:14:02.047 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:14:02.047 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:02.047 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:14:02.047 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:14:02.047 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:02.047 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:14:02.047 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:14:02.047 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:14:02.047 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:14:02.047 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:02.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.045+0000 7f99881cb640 1 librados: starting msgr at 2024-03-25T18:14:02.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.045+0000 7f99881cb640 1 librados: starting objecter 2024-03-25T18:14:02.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.045+0000 7f99881cb640 1 Processor -- start 2024-03-25T18:14:02.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.045+0000 7f99881cb640 1 -- start start 2024-03-25T18:14:02.047 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.045+0000 7f99881cb640 1 librados: setting wanted keys 2024-03-25T18:14:02.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.045+0000 7f99881cb640 1 librados: calling monclient init 2024-03-25T18:14:02.048 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.045+0000 7f99881cb640 10 monclient: init 2024-03-25T18:14:02.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.046+0000 7f99881cb640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:02.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.046+0000 7f99881cb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:02.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.046+0000 7f99881cb640 10 monclient(hunting): picked mon.noname-c con 0x7f99800a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:14:02.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.046+0000 7f99881cb640 10 monclient(hunting): picked mon.noname-a con 0x7f998013f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:14:02.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.046+0000 7f99881cb640 10 monclient(hunting): picked mon.noname-b con 0x7f99801430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:14:02.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.046+0000 7f99881cb640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99801d7380 con 0x7f998013f710 2024-03-25T18:14:02.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.046+0000 7f99881cb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9980139840 con 0x7f99801430c0 2024-03-25T18:14:02.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.046+0000 7f99881cb640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99801d8200 con 0x7f99800a98f0 2024-03-25T18:14:02.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.046+0000 7f99881cb640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:02.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.046+0000 7f99881cb640 10 monclient(hunting): authenticate will time out at 7228.284846s 2024-03-25T18:14:02.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.052+0000 7f9985766640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f998013f710 0x7f99801da880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:57146/0 (socket says 172.21.15.64:57146) 2024-03-25T18:14:02.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.052+0000 7f9985766640 1 -- 172.21.15.64:0/3211849690 learned_addr learned my addr 172.21.15.64:0/3211849690 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:14:02.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.054+0000 7f99767fc640 10 client.?.objecter ms_handle_connect 0x7f998013f710 2024-03-25T18:14:02.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.054+0000 7f99767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:02.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.063+0000 7f99767fc640 10 client.?.objecter ms_handle_connect 0x7f99801430c0 2024-03-25T18:14:02.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.063+0000 7f99767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:02.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.063+0000 7f99767fc640 10 client.?.objecter ms_handle_connect 0x7f99800a98f0 2024-03-25T18:14:02.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.063+0000 7f99767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:02.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.067+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596510372 0 0) 0x7f99801d7380 con 0x7f998013f710 2024-03-25T18:14:02.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.067+0000 7f99767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:02.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.067+0000 7f99767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:02.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.067+0000 7f99767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:02.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.067+0000 7f99767fc640 10 monclient(hunting): my global_id is 4425 2024-03-25T18:14:02.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.067+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9964004eb0 con 0x7f998013f710 2024-03-25T18:14:02.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.074+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418986727 0 0) 0x7f99801d8200 con 0x7f99800a98f0 2024-03-25T18:14:02.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.074+0000 7f99767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:02.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.074+0000 7f99767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:02.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.074+0000 7f99767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:02.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.074+0000 7f99767fc640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:14:02.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.074+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99801d7380 con 0x7f99800a98f0 2024-03-25T18:14:02.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.079+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081943304 0 0) 0x7f9980139840 con 0x7f99801430c0 2024-03-25T18:14:02.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.079+0000 7f99767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:02.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.079+0000 7f99767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:02.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.079+0000 7f99767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:02.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.079+0000 7f99767fc640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:14:02.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.079+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99801d8200 con 0x7f99801430c0 2024-03-25T18:14:02.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.085+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3515312569 0 0) 0x7f9964004eb0 con 0x7f998013f710 2024-03-25T18:14:02.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.085+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9980139840 con 0x7f998013f710 2024-03-25T18:14:02.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.085+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9970001a10 con 0x7f998013f710 2024-03-25T18:14:02.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.085+0000 7f99767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:02.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.085+0000 7f99767fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:02.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.085+0000 7f99767fc640 10 monclient(hunting): dump: 2024-03-25T18:14:02.087 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:02.087 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:02.087 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:02.087 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:02.087 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:02.087 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:02.087 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:02.087 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:02.087 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:02.087 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:02.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.085+0000 7f99767fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:02.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.097+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1932440032 0 0) 0x7f99801d8200 con 0x7f99801430c0 2024-03-25T18:14:02.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.097+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9964004eb0 con 0x7f99801430c0 2024-03-25T18:14:02.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.097+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9978001bd0 con 0x7f99801430c0 2024-03-25T18:14:02.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.097+0000 7f99767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:02.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.097+0000 7f99767fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:02.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.097+0000 7f99767fc640 10 monclient(hunting): dump: 2024-03-25T18:14:02.099 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:02.099 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:02.099 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:02.099 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:02.099 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:02.099 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:02.099 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:02.099 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:02.099 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:02.099 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:02.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3699542133 0 0) 0x7f9980139840 con 0x7f998013f710 2024-03-25T18:14:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 >> v1:172.21.15.64:6790/0 conn(0x7f99800a98f0 legacy=0x7f99801d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 >> v1:172.21.15.86:6789/0 conn(0x7f99801430c0 legacy=0x7f99801daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99767fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99767fc640 1 monclient: found mon.a 2024-03-25T18:14:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99767fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:14:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99801d75d0 con 0x7f998013f710 2024-03-25T18:14:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99767fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99767fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99881cb640 5 monclient: authenticate success, global_id 4425 2024-03-25T18:14:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99881cb640 10 monclient: _renew_subs 2024-03-25T18:14:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99881cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:14:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99881cb640 1 -- 172.21.15.64:0/3211849690 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99801db6a0 con 0x7f998013f710 2024-03-25T18:14:02.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99881cb640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:02.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99881cb640 10 monclient: _renew_subs 2024-03-25T18:14:02.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99881cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:14:02.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99881cb640 1 -- 172.21.15.64:0/3211849690 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9980137910 con 0x7f998013f710 2024-03-25T18:14:02.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.099+0000 7f99881cb640 1 librados: init done 2024-03-25T18:14:02.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.100+0000 7f99881cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:02.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.100+0000 7f99881cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:02.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.100+0000 7f99881cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:14:02.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.100+0000 7f99881cb640 1 -- 172.21.15.64:0/3211849690 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99800627b0 con 0x7f998013f710 2024-03-25T18:14:02.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.416+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9970001c70 con 0x7f998013f710 2024-03-25T18:14:02.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.416+0000 7f99767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:02.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.416+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9970003a60 con 0x7f998013f710 2024-03-25T18:14:02.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.416+0000 7f99767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:02.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.416+0000 7f99767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:02.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.416+0000 7f99767fc640 10 monclient: dump: 2024-03-25T18:14:02.417 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:02.417 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:02.417 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:02.417 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:02.418 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:02.418 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:02.418 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:02.418 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:02.418 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:02.418 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:02.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.416+0000 7f9977fff640 4 set_mon_vals no callback set 2024-03-25T18:14:02.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.417+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f9970003bc0 con 0x7f998013f710 2024-03-25T18:14:02.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.417+0000 7f9985f67640 10 monclient: get_auth_request con 0x7f996406df70 auth_method 0 2024-03-25T18:14:02.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.417+0000 7f99767fc640 10 client.4425.objecter ms_handle_connect 0x7f996406df70 2024-03-25T18:14:02.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.423+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f99700548e0 con 0x7f998013f710 2024-03-25T18:14:02.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.423+0000 7f99767fc640 10 client.4425.objecter ms_dispatch 0x7f9980000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:02.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.423+0000 7f99767fc640 3 client.4425.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:02.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.423+0000 7f99767fc640 3 client.4425.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:02.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.423+0000 7f99767fc640 20 client.4425.objecter dump_active .. 0 homeless 2024-03-25T18:14:02.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.423+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f99700503f0 con 0x7f998013f710 2024-03-25T18:14:02.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.423+0000 7f99767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:02.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.423+0000 7f99767fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:02.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.684+0000 7f99881cb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:02.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.685+0000 7f99881cb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:02.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.685+0000 7f99881cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:14:02.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.685+0000 7f99881cb640 1 -- 172.21.15.64:0/3211849690 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9980069f40 con 0x7f998013f710 2024-03-25T18:14:02.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.727+0000 7f99767fc640 1 -- 172.21.15.64:0/3211849690 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f9970052320 con 0x7f998013f710 2024-03-25T18:14:02.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.727+0000 7f99767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:02.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.727+0000 7f99767fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:02.728 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:14:02.728 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:02.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.729+0000 7f99881cb640 10 librados: watch_flush enter 2024-03-25T18:14:02.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.729+0000 7f99881cb640 10 librados: watch_flush exit 2024-03-25T18:14:02.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.729+0000 7f99881cb640 20 client.4425.objecter shutdown clearing up homeless session... 2024-03-25T18:14:02.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.729+0000 7f99881cb640 10 client.4425.objecter successfully canceled tick 2024-03-25T18:14:02.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.730+0000 7f99881cb640 1 -- 172.21.15.64:0/3211849690 >> v1:172.21.15.64:6800/2915543638 conn(0x7f996406df70 legacy=0x7f9964070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:02.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.730+0000 7f99881cb640 10 monclient: shutdown 2024-03-25T18:14:02.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.730+0000 7f99881cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:02.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.730+0000 7f99881cb640 1 -- 172.21.15.64:0/3211849690 >> v1:172.21.15.64:6789/0 conn(0x7f998013f710 legacy=0x7f99801da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:02.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.730+0000 7f99881cb640 1 -- 172.21.15.64:0/3211849690 shutdown_connections 2024-03-25T18:14:02.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.730+0000 7f99881cb640 1 -- 172.21.15.64:0/3211849690 >> 172.21.15.64:0/3211849690 conn(0x7f99800b7ea0 msgr2=0x7f9980074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:02.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.730+0000 7f99881cb640 1 -- 172.21.15.64:0/3211849690 shutdown_connections 2024-03-25T18:14:02.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.730+0000 7f99881cb640 1 -- 172.21.15.64:0/3211849690 wait complete. 2024-03-25T18:14:02.732 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:02.730+0000 7f99881cb640 1 librados: shutdown 2024-03-25T18:14:02.741 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:09.548 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:09.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.663+0000 7f47dea38640 10 monclient: get_monmap_and_config 2024-03-25T18:14:09.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.663+0000 7f47dea38640 10 monclient: build_initial_monmap 2024-03-25T18:14:09.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.663+0000 7f47dea38640 10 monclient: monmap: 2024-03-25T18:14:09.664 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:14:09.665 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:09.665 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:14:09.664568+0000 2024-03-25T18:14:09.665 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:14:09.664568+0000 2024-03-25T18:14:09.665 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:09.665 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:09.665 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:14:09.665 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:14:09.665 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:14:09.665 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:09.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.664+0000 7f47dea38640 1 Processor -- start 2024-03-25T18:14:09.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.664+0000 7f47dea38640 1 -- start start 2024-03-25T18:14:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.664+0000 7f47dea38640 10 monclient: init 2024-03-25T18:14:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.665+0000 7f47dea38640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.665+0000 7f47dea38640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.665+0000 7f47dea38640 10 monclient(hunting): picked mon.noname-b con 0x7f47d80a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:14:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.665+0000 7f47dea38640 10 monclient(hunting): picked mon.noname-c con 0x7f47d813f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:14:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.665+0000 7f47dea38640 10 monclient(hunting): picked mon.noname-a con 0x7f47d81430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:14:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.665+0000 7f47dea38640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d8142520 con 0x7f47d81430c0 2024-03-25T18:14:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.665+0000 7f47dea38640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d80b28a0 con 0x7f47d813f710 2024-03-25T18:14:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.665+0000 7f47dea38640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d80b2a30 con 0x7f47d80a98f0 2024-03-25T18:14:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.665+0000 7f47dea38640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:09.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.665+0000 7f47dea38640 10 monclient(hunting): authenticate will time out at 7235.904177s 2024-03-25T18:14:09.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.680+0000 7f47d77fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f47d813f710 0x7f47d81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:33856/0 (socket says 172.21.15.64:33856) 2024-03-25T18:14:09.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.680+0000 7f47d77fe640 1 -- 172.21.15.64:0/3513601143 learned_addr learned my addr 172.21.15.64:0/3513601143 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:14:09.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.691+0000 7f47d6ffd640 1 -- 172.21.15.64:0/3513601143 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695710872 0 0) 0x7f47d80b28a0 con 0x7f47d813f710 2024-03-25T18:14:09.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.692+0000 7f47d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.692+0000 7f47d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.692+0000 7f47d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.692+0000 7f47d6ffd640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:14:09.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.692+0000 7f47d6ffd640 1 -- 172.21.15.64:0/3513601143 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c0003520 con 0x7f47d813f710 2024-03-25T18:14:09.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.705+0000 7f47d6ffd640 1 -- 172.21.15.64:0/3513601143 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826024340 0 0) 0x7f47d8142520 con 0x7f47d81430c0 2024-03-25T18:14:09.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.705+0000 7f47d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.705+0000 7f47d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.705+0000 7f47d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.705+0000 7f47d6ffd640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:14:09.706 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.705+0000 7f47d6ffd640 1 -- 172.21.15.64:0/3513601143 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47d80b28a0 con 0x7f47d81430c0 2024-03-25T18:14:09.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.710+0000 7f47d6ffd640 1 -- 172.21.15.64:0/3513601143 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 389271529 0 0) 0x7f47c0003520 con 0x7f47d813f710 2024-03-25T18:14:09.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.710+0000 7f47d6ffd640 1 -- 172.21.15.64:0/3513601143 >> v1:172.21.15.64:6790/0 conn(0x7f47d80a98f0 legacy=0x7f47d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.710+0000 7f47d6ffd640 1 -- 172.21.15.64:0/3513601143 >> v1:172.21.15.64:6789/0 conn(0x7f47d81430c0 legacy=0x7f47d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.710+0000 7f47d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:09.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.710+0000 7f47d6ffd640 1 monclient: found mon.noname-c 2024-03-25T18:14:09.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.710+0000 7f47d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:09.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.710+0000 7f47d6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:14:09.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.710+0000 7f47d6ffd640 1 -- 172.21.15.64:0/3513601143 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d80b44c0 con 0x7f47d813f710 2024-03-25T18:14:09.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.710+0000 7f47d6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:09.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.710+0000 7f47d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:09.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.710+0000 7f47d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:09.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.710+0000 7f47d6ffd640 1 -- 172.21.15.64:0/3513601143 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f47cc001690 con 0x7f47d813f710 2024-03-25T18:14:09.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.710+0000 7f47dea38640 5 monclient: authenticate success, global_id 4423 2024-03-25T18:14:09.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.710+0000 7f47dea38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:09.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.710+0000 7f47d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.710+0000 7f47d6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:09.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.710+0000 7f47d6ffd640 10 monclient: dump: 2024-03-25T18:14:09.712 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:09.712 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:09.712 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:09.712 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:09.712 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.712 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:09.712 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:09.712 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:09.712 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:09.712 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:09.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.710+0000 7f47dea38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:09.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.717+0000 7f47d6ffd640 1 -- 172.21.15.64:0/3513601143 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47cc0017f0 con 0x7f47d813f710 2024-03-25T18:14:09.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.717+0000 7f47d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:09.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.717+0000 7f47dea38640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:09.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.717+0000 7f47d6ffd640 1 -- 172.21.15.64:0/3513601143 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f47cc001c00 con 0x7f47d813f710 2024-03-25T18:14:09.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.717+0000 7f47d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.717+0000 7f47dea38640 4 set_mon_vals no callback set 2024-03-25T18:14:09.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.717+0000 7f47d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:09.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.717+0000 7f47d6ffd640 10 monclient: dump: 2024-03-25T18:14:09.718 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:09.719 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:09.719 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:09.719 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:09.719 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.719 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:09.719 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:09.719 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:09.719 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:09.719 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:09.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.718+0000 7f47dea38640 10 monclient: shutdown 2024-03-25T18:14:09.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.718+0000 7f47dea38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:09.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.718+0000 7f47dea38640 1 -- 172.21.15.64:0/3513601143 >> v1:172.21.15.86:6789/0 conn(0x7f47d813f710 legacy=0x7f47d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.718+0000 7f47dea38640 1 -- 172.21.15.64:0/3513601143 shutdown_connections 2024-03-25T18:14:09.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.718+0000 7f47dea38640 1 -- 172.21.15.64:0/3513601143 >> 172.21.15.64:0/3513601143 conn(0x7f47d80b7ea0 msgr2=0x7f47d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:09.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.718+0000 7f47dea38640 1 -- 172.21.15.64:0/3513601143 shutdown_connections 2024-03-25T18:14:09.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.718+0000 7f47dea38640 1 -- 172.21.15.64:0/3513601143 wait complete. 2024-03-25T18:14:09.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.718+0000 7f47dea38640 10 monclient: build_initial_monmap 2024-03-25T18:14:09.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.718+0000 7f47dea38640 10 monclient: monmap: 2024-03-25T18:14:09.721 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:14:09.721 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:09.721 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:14:09.721 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:14:09.722 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:09.722 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:14:09.722 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:14:09.722 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:14:09.722 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:14:09.722 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:09.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.719+0000 7f47dea38640 1 librados: starting msgr at 2024-03-25T18:14:09.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.719+0000 7f47dea38640 1 librados: starting objecter 2024-03-25T18:14:09.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.719+0000 7f47dea38640 1 Processor -- start 2024-03-25T18:14:09.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.719+0000 7f47dea38640 1 -- start start 2024-03-25T18:14:09.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.719+0000 7f47dea38640 1 librados: setting wanted keys 2024-03-25T18:14:09.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.719+0000 7f47dea38640 1 librados: calling monclient init 2024-03-25T18:14:09.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.719+0000 7f47dea38640 10 monclient: init 2024-03-25T18:14:09.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.720+0000 7f47dea38640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:09.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.720+0000 7f47dea38640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:09.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.720+0000 7f47dea38640 10 monclient(hunting): picked mon.noname-a con 0x7f47d80a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:14:09.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.720+0000 7f47dea38640 10 monclient(hunting): picked mon.noname-b con 0x7f47d813f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:14:09.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.720+0000 7f47dea38640 10 monclient(hunting): picked mon.noname-c con 0x7f47d81430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:14:09.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.720+0000 7f47dea38640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d8139870 con 0x7f47d80a98f0 2024-03-25T18:14:09.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.720+0000 7f47dea38640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d81d51f0 con 0x7f47d813f710 2024-03-25T18:14:09.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.720+0000 7f47dea38640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d8138830 con 0x7f47d81430c0 2024-03-25T18:14:09.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.720+0000 7f47dea38640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:09.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.720+0000 7f47dea38640 10 monclient(hunting): authenticate will time out at 7235.958678s 2024-03-25T18:14:09.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.729+0000 7f47dcfd5640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f47d81430c0 0x7f47d81e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:59524/0 (socket says 172.21.15.64:59524) 2024-03-25T18:14:09.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.729+0000 7f47dcfd5640 1 -- 172.21.15.64:0/996240278 learned_addr learned my addr 172.21.15.64:0/996240278 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:14:09.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.736+0000 7f47d4ff9640 10 client.?.objecter ms_handle_connect 0x7f47d81430c0 2024-03-25T18:14:09.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.736+0000 7f47d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:09.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.738+0000 7f47d4ff9640 10 client.?.objecter ms_handle_connect 0x7f47d80a98f0 2024-03-25T18:14:09.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.738+0000 7f47d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:09.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.740+0000 7f47d4ff9640 10 client.?.objecter ms_handle_connect 0x7f47d813f710 2024-03-25T18:14:09.741 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.740+0000 7f47d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:09.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.745+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764634507 0 0) 0x7f47d8138830 con 0x7f47d81430c0 2024-03-25T18:14:09.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.745+0000 7f47d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.745+0000 7f47d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.745+0000 7f47d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.745+0000 7f47d4ff9640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:14:09.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.745+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c4004eb0 con 0x7f47d81430c0 2024-03-25T18:14:09.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.748+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069124764 0 0) 0x7f47d8139870 con 0x7f47d80a98f0 2024-03-25T18:14:09.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.748+0000 7f47d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.748+0000 7f47d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.748+0000 7f47d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.748+0000 7f47d4ff9640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:14:09.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.748+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47d8138830 con 0x7f47d80a98f0 2024-03-25T18:14:09.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.762+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290235990 0 0) 0x7f47d81d51f0 con 0x7f47d813f710 2024-03-25T18:14:09.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.762+0000 7f47d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.762+0000 7f47d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.762+0000 7f47d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.762+0000 7f47d4ff9640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:14:09.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.762+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47d8139870 con 0x7f47d813f710 2024-03-25T18:14:09.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.763+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 441873188 0 0) 0x7f47c4004eb0 con 0x7f47d81430c0 2024-03-25T18:14:09.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.763+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47d81d51f0 con 0x7f47d81430c0 2024-03-25T18:14:09.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.763+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f47d0001c90 con 0x7f47d81430c0 2024-03-25T18:14:09.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.763+0000 7f47d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.763+0000 7f47d4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:09.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.763+0000 7f47d4ff9640 10 monclient(hunting): dump: 2024-03-25T18:14:09.765 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:09.765 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:09.765 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:09.765 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:09.765 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.765 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:09.765 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:09.765 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:09.765 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:09.765 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:09.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.763+0000 7f47d4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:09.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.764+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1257055866 0 0) 0x7f47d8138830 con 0x7f47d80a98f0 2024-03-25T18:14:09.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.764+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47c4004eb0 con 0x7f47d80a98f0 2024-03-25T18:14:09.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.764+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f47c80018b0 con 0x7f47d80a98f0 2024-03-25T18:14:09.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.764+0000 7f47d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.764+0000 7f47d4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:09.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.764+0000 7f47d4ff9640 10 monclient(hunting): dump: 2024-03-25T18:14:09.766 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:09.766 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:09.766 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:09.766 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:09.766 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.767 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:09.767 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:09.767 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:09.767 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:09.767 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:09.776 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.775+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1764592616 0 0) 0x7f47c4004eb0 con 0x7f47d80a98f0 2024-03-25T18:14:09.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.775+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 >> v1:172.21.15.64:6790/0 conn(0x7f47d81430c0 legacy=0x7f47d81e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.775+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 >> v1:172.21.15.86:6789/0 conn(0x7f47d813f710 legacy=0x7f47d81d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.775+0000 7f47d4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:14:09.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.775+0000 7f47d4ff9640 1 monclient: found mon.a 2024-03-25T18:14:09.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.775+0000 7f47d4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:09.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.775+0000 7f47d4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:14:09.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.775+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d81d77a0 con 0x7f47d80a98f0 2024-03-25T18:14:09.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.775+0000 7f47d4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:14:09.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.775+0000 7f47d4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:14:09.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.775+0000 7f47d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:09.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.775+0000 7f47dea38640 5 monclient: authenticate success, global_id 4431 2024-03-25T18:14:09.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.775+0000 7f47dea38640 10 monclient: _renew_subs 2024-03-25T18:14:09.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.775+0000 7f47dea38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:14:09.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.775+0000 7f47dea38640 1 -- 172.21.15.64:0/996240278 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d81d83d0 con 0x7f47d80a98f0 2024-03-25T18:14:09.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.776+0000 7f47dea38640 10 client.4431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:09.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.776+0000 7f47dea38640 10 monclient: _renew_subs 2024-03-25T18:14:09.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.776+0000 7f47dea38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:14:09.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.776+0000 7f47dea38640 1 -- 172.21.15.64:0/996240278 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d8138c80 con 0x7f47d80a98f0 2024-03-25T18:14:09.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.776+0000 7f47dea38640 1 librados: init done 2024-03-25T18:14:09.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.776+0000 7f47dea38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:09.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.776+0000 7f47dea38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:09.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.776+0000 7f47dea38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:14:09.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:09.776+0000 7f47dea38640 1 -- 172.21.15.64:0/996240278 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d80b3e80 con 0x7f47d80a98f0 2024-03-25T18:14:10.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.092+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47c8003610 con 0x7f47d80a98f0 2024-03-25T18:14:10.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.092+0000 7f47d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:10.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.092+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f47c8003920 con 0x7f47d80a98f0 2024-03-25T18:14:10.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.092+0000 7f47d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:10.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.092+0000 7f47d4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:10.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.092+0000 7f47d4ff9640 10 monclient: dump: 2024-03-25T18:14:10.093 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:10.094 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:10.094 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:10.094 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:10.094 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:10.094 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:10.094 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:10.094 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:10.094 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:10.094 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:10.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.092+0000 7f47d67fc640 4 set_mon_vals no callback set 2024-03-25T18:14:10.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.092+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f47c80198f0 con 0x7f47d80a98f0 2024-03-25T18:14:10.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.093+0000 7f47d77fe640 10 monclient: get_auth_request con 0x7f47c406df70 auth_method 0 2024-03-25T18:14:10.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.093+0000 7f47d4ff9640 10 client.4431.objecter ms_handle_connect 0x7f47c406df70 2024-03-25T18:14:10.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.098+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f47c80535c0 con 0x7f47d80a98f0 2024-03-25T18:14:10.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.098+0000 7f47d4ff9640 10 client.4431.objecter ms_dispatch 0x7f47d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:10.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.098+0000 7f47d4ff9640 3 client.4431.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:10.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.098+0000 7f47d4ff9640 3 client.4431.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:10.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.098+0000 7f47d4ff9640 20 client.4431.objecter dump_active .. 0 homeless 2024-03-25T18:14:10.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.098+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f47c804f0d0 con 0x7f47d80a98f0 2024-03-25T18:14:10.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.098+0000 7f47d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:10.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.098+0000 7f47d4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:10.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.333+0000 7f47dea38640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:10.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.333+0000 7f47dea38640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:10.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.333+0000 7f47dea38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:14:10.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.333+0000 7f47dea38640 1 -- 172.21.15.64:0/996240278 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47d81d7930 con 0x7f47d80a98f0 2024-03-25T18:14:10.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.356+0000 7f47d4ff9640 1 -- 172.21.15.64:0/996240278 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f47c8051000 con 0x7f47d80a98f0 2024-03-25T18:14:10.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.356+0000 7f47d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:10.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.356+0000 7f47d4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:10.357 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:14:10.358 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:10.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.359+0000 7f47dea38640 10 librados: watch_flush enter 2024-03-25T18:14:10.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.359+0000 7f47dea38640 10 librados: watch_flush exit 2024-03-25T18:14:10.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.359+0000 7f47dea38640 20 client.4431.objecter shutdown clearing up homeless session... 2024-03-25T18:14:10.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.359+0000 7f47dea38640 10 client.4431.objecter successfully canceled tick 2024-03-25T18:14:10.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.359+0000 7f47dea38640 1 -- 172.21.15.64:0/996240278 >> v1:172.21.15.64:6800/2915543638 conn(0x7f47c406df70 legacy=0x7f47c4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:10.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.359+0000 7f47dea38640 10 monclient: shutdown 2024-03-25T18:14:10.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.359+0000 7f47dea38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:10.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.359+0000 7f47dea38640 1 -- 172.21.15.64:0/996240278 >> v1:172.21.15.64:6789/0 conn(0x7f47d80a98f0 legacy=0x7f47d8138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:10.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.359+0000 7f47dea38640 1 -- 172.21.15.64:0/996240278 shutdown_connections 2024-03-25T18:14:10.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.359+0000 7f47dea38640 1 -- 172.21.15.64:0/996240278 >> 172.21.15.64:0/996240278 conn(0x7f47d80b7ea0 msgr2=0x7f47d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:10.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.359+0000 7f47dea38640 1 -- 172.21.15.64:0/996240278 shutdown_connections 2024-03-25T18:14:10.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.359+0000 7f47dea38640 1 -- 172.21.15.64:0/996240278 wait complete. 2024-03-25T18:14:10.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:10.359+0000 7f47dea38640 1 librados: shutdown 2024-03-25T18:14:10.372 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:17.178 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:17.200 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:05:00.312 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:17.277 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:05:00.389 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:17.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.289+0000 7f682984a640 10 monclient: get_monmap_and_config 2024-03-25T18:14:17.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.289+0000 7f682984a640 10 monclient: build_initial_monmap 2024-03-25T18:14:17.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.289+0000 7f682984a640 10 monclient: monmap: 2024-03-25T18:14:17.291 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:14:17.291 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:17.291 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:14:17.291473+0000 2024-03-25T18:14:17.291 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:14:17.291473+0000 2024-03-25T18:14:17.291 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:17.291 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:17.292 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:14:17.292 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:14:17.292 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:14:17.292 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:17.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.290+0000 7f682984a640 1 Processor -- start 2024-03-25T18:14:17.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.290+0000 7f682984a640 1 -- start start 2024-03-25T18:14:17.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.290+0000 7f682984a640 10 monclient: init 2024-03-25T18:14:17.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.290+0000 7f682984a640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:17.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.290+0000 7f682984a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:17.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.290+0000 7f682984a640 10 monclient(hunting): picked mon.noname-b con 0x7f68240b4d60 addr v1:172.21.15.64:6790/0 2024-03-25T18:14:17.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.290+0000 7f682984a640 10 monclient(hunting): picked mon.noname-c con 0x7f68240b1730 addr v1:172.21.15.86:6789/0 2024-03-25T18:14:17.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.290+0000 7f682984a640 10 monclient(hunting): picked mon.noname-a con 0x7f682414be90 addr v1:172.21.15.64:6789/0 2024-03-25T18:14:17.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.291+0000 7f682984a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68240af940 con 0x7f682414be90 2024-03-25T18:14:17.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.291+0000 7f682984a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68240b0a70 con 0x7f68240b1730 2024-03-25T18:14:17.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.291+0000 7f682984a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68240b0d50 con 0x7f68240b4d60 2024-03-25T18:14:17.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.291+0000 7f682984a640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:17.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.291+0000 7f682984a640 10 monclient(hunting): authenticate will time out at 7243.530707s 2024-03-25T18:14:17.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.307+0000 7f6822ffd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f68240b4d60 0x7f682413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:33774/0 (socket says 172.21.15.64:33774) 2024-03-25T18:14:17.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.307+0000 7f6822ffd640 1 -- 172.21.15.64:0/702631849 learned_addr learned my addr 172.21.15.64:0/702631849 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:14:17.323 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:05:00.436 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:17.324 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:05:00.402 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:17.327 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:05:00.440 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:17.328 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:05:00.440 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:17.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.328+0000 7f68227fc640 1 -- 172.21.15.64:0/702631849 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776066769 0 0) 0x7f68240b0d50 con 0x7f68240b4d60 2024-03-25T18:14:17.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.328+0000 7f68227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:17.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.328+0000 7f68227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:17.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.329+0000 7f68227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:17.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.329+0000 7f68227fc640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:14:17.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.329+0000 7f68227fc640 1 -- 172.21.15.64:0/702631849 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6810003520 con 0x7f68240b4d60 2024-03-25T18:14:17.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.334+0000 7f68227fc640 1 -- 172.21.15.64:0/702631849 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435695828 0 0) 0x7f68240b0a70 con 0x7f68240b1730 2024-03-25T18:14:17.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.334+0000 7f68227fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:17.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.334+0000 7f68227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:17.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.334+0000 7f68227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:17.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.334+0000 7f68227fc640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:14:17.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.334+0000 7f68227fc640 1 -- 172.21.15.64:0/702631849 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68240b0d50 con 0x7f68240b1730 2024-03-25T18:14:17.338 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:05:00.416 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:17.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.350+0000 7f68227fc640 1 -- 172.21.15.64:0/702631849 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1673262403 0 0) 0x7f6810003520 con 0x7f68240b4d60 2024-03-25T18:14:17.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.350+0000 7f68227fc640 1 -- 172.21.15.64:0/702631849 >> v1:172.21.15.86:6789/0 conn(0x7f68240b1730 legacy=0x7f68240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.351+0000 7f68227fc640 1 -- 172.21.15.64:0/702631849 >> v1:172.21.15.64:6789/0 conn(0x7f682414be90 legacy=0x7f682414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.351+0000 7f68227fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:17.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.351+0000 7f68227fc640 1 monclient: found mon.noname-b 2024-03-25T18:14:17.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.351+0000 7f68227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:17.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.351+0000 7f68227fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:14:17.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.351+0000 7f68227fc640 1 -- 172.21.15.64:0/702631849 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68240b1030 con 0x7f68240b4d60 2024-03-25T18:14:17.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.351+0000 7f68227fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:17.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.351+0000 7f68227fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:17.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.351+0000 7f68227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:17.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.351+0000 7f68227fc640 1 -- 172.21.15.64:0/702631849 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f6814001690 con 0x7f68240b4d60 2024-03-25T18:14:17.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.351+0000 7f682984a640 5 monclient: authenticate success, global_id 4436 2024-03-25T18:14:17.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.351+0000 7f682984a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:17.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.351+0000 7f68227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:17.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.351+0000 7f68227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:17.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.351+0000 7f68227fc640 10 monclient: dump: 2024-03-25T18:14:17.354 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:17.354 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:17.354 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:17.354 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:17.354 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:17.354 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:17.354 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:17.354 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:17.354 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:17.354 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:17.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.351+0000 7f682984a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:17.358 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:05:00.471 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:17.373 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:05:00.452 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:17.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.377+0000 7f68227fc640 1 -- 172.21.15.64:0/702631849 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68140017f0 con 0x7f68240b4d60 2024-03-25T18:14:17.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.377+0000 7f68227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:17.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.377+0000 7f68227fc640 1 -- 172.21.15.64:0/702631849 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f6814001c00 con 0x7f68240b4d60 2024-03-25T18:14:17.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.377+0000 7f68227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:17.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.377+0000 7f68227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:17.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.377+0000 7f68227fc640 10 monclient: dump: 2024-03-25T18:14:17.379 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:17.379 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:17.379 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:17.379 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:17.379 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:17.379 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:17.379 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:17.379 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:17.379 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:17.380 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:17.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.377+0000 7f682984a640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:17.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.377+0000 7f682984a640 4 set_mon_vals no callback set 2024-03-25T18:14:17.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.377+0000 7f682984a640 10 monclient: shutdown 2024-03-25T18:14:17.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.377+0000 7f682984a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:17.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.377+0000 7f682984a640 1 -- 172.21.15.64:0/702631849 >> v1:172.21.15.64:6790/0 conn(0x7f68240b4d60 legacy=0x7f682413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.377+0000 7f682984a640 1 -- 172.21.15.64:0/702631849 shutdown_connections 2024-03-25T18:14:17.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.377+0000 7f682984a640 1 -- 172.21.15.64:0/702631849 >> 172.21.15.64:0/702631849 conn(0x7f68240617a0 msgr2=0x7f6824061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:17.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.378+0000 7f682984a640 1 -- 172.21.15.64:0/702631849 shutdown_connections 2024-03-25T18:14:17.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.378+0000 7f682984a640 1 -- 172.21.15.64:0/702631849 wait complete. 2024-03-25T18:14:17.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.378+0000 7f682984a640 10 monclient: build_initial_monmap 2024-03-25T18:14:17.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.378+0000 7f682984a640 10 monclient: monmap: 2024-03-25T18:14:17.382 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:14:17.382 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:17.382 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:14:17.382 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:14:17.382 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:17.382 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:14:17.382 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:14:17.382 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:14:17.382 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:14:17.383 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:17.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.378+0000 7f682984a640 1 librados: starting msgr at 2024-03-25T18:14:17.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.378+0000 7f682984a640 1 librados: starting objecter 2024-03-25T18:14:17.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.379+0000 7f682984a640 1 Processor -- start 2024-03-25T18:14:17.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.379+0000 7f682984a640 1 -- start start 2024-03-25T18:14:17.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.379+0000 7f682984a640 1 librados: setting wanted keys 2024-03-25T18:14:17.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.379+0000 7f682984a640 1 librados: calling monclient init 2024-03-25T18:14:17.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.379+0000 7f682984a640 10 monclient: init 2024-03-25T18:14:17.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.379+0000 7f682984a640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:17.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.379+0000 7f682984a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:17.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.379+0000 7f682984a640 10 monclient(hunting): picked mon.noname-a con 0x7f68240b1730 addr v1:172.21.15.64:6789/0 2024-03-25T18:14:17.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.379+0000 7f682984a640 10 monclient(hunting): picked mon.noname-c con 0x7f68240b4d60 addr v1:172.21.15.64:6790/0 2024-03-25T18:14:17.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.379+0000 7f682984a640 10 monclient(hunting): picked mon.noname-b con 0x7f682414be90 addr v1:172.21.15.86:6789/0 2024-03-25T18:14:17.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.379+0000 7f682984a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6824138010 con 0x7f68240b1730 2024-03-25T18:14:17.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.379+0000 7f682984a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68241382f0 con 0x7f682414be90 2024-03-25T18:14:17.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.379+0000 7f682984a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6824136d90 con 0x7f68240b4d60 2024-03-25T18:14:17.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.379+0000 7f682984a640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:17.384 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.379+0000 7f682984a640 10 monclient(hunting): authenticate will time out at 7243.619219s 2024-03-25T18:14:17.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.417+0000 7f68237fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f682414be90 0x7f68241427d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:40442/0 (socket says 172.21.15.64:40442) 2024-03-25T18:14:17.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.417+0000 7f68237fe640 1 -- 172.21.15.64:0/2393381269 learned_addr learned my addr 172.21.15.64:0/2393381269 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:14:17.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.423+0000 7f6820ff9640 10 client.?.objecter ms_handle_connect 0x7f682414be90 2024-03-25T18:14:17.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.423+0000 7f6820ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:17.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.440+0000 7f6820ff9640 10 client.?.objecter ms_handle_connect 0x7f68240b4d60 2024-03-25T18:14:17.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.440+0000 7f6820ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:17.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.443+0000 7f6820ff9640 10 client.?.objecter ms_handle_connect 0x7f68240b1730 2024-03-25T18:14:17.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.443+0000 7f6820ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:17.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.445+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009051659 0 0) 0x7f68241382f0 con 0x7f682414be90 2024-03-25T18:14:17.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.446+0000 7f6820ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:17.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.446+0000 7f6820ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:17.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.446+0000 7f6820ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:17.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.446+0000 7f6820ff9640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:14:17.448 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.446+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6804004eb0 con 0x7f682414be90 2024-03-25T18:14:17.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.468+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803130266 0 0) 0x7f6824136d90 con 0x7f68240b4d60 2024-03-25T18:14:17.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.468+0000 7f6820ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:17.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.468+0000 7f6820ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:17.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.468+0000 7f6820ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:17.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.468+0000 7f6820ff9640 10 monclient(hunting): my global_id is 4439 2024-03-25T18:14:17.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.468+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68241382f0 con 0x7f68240b4d60 2024-03-25T18:14:17.475 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.473+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 351411855 0 0) 0x7f6804004eb0 con 0x7f682414be90 2024-03-25T18:14:17.475 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.473+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6824136d90 con 0x7f682414be90 2024-03-25T18:14:17.475 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.473+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f681c001c30 con 0x7f682414be90 2024-03-25T18:14:17.475 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.473+0000 7f6820ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:17.475 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.473+0000 7f6820ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:17.475 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.473+0000 7f6820ff9640 10 monclient(hunting): dump: 2024-03-25T18:14:17.475 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:17.475 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:17.475 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:17.475 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:17.475 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:17.475 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:17.475 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:17.476 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:17.476 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:17.476 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:17.476 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.473+0000 7f6820ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:17.482 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.480+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548585662 0 0) 0x7f6824138010 con 0x7f68240b1730 2024-03-25T18:14:17.482 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.481+0000 7f6820ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:17.482 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.481+0000 7f6820ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:17.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.481+0000 7f6820ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:17.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.481+0000 7f6820ff9640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:14:17.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.481+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6804004eb0 con 0x7f68240b1730 2024-03-25T18:14:17.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1535168349 0 0) 0x7f6824136d90 con 0x7f682414be90 2024-03-25T18:14:17.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 >> v1:172.21.15.64:6790/0 conn(0x7f68240b4d60 legacy=0x7f6824143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 >> v1:172.21.15.64:6789/0 conn(0x7f68240b1730 legacy=0x7f6824135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f6820ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:14:17.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f6820ff9640 1 monclient: found mon.b 2024-03-25T18:14:17.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f6820ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:17.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f6820ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:14:17.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6824144010 con 0x7f682414be90 2024-03-25T18:14:17.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f6820ff9640 10 monclient: _finish_auth 0 2024-03-25T18:14:17.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f6820ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:14:17.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f6820ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:17.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f682984a640 5 monclient: authenticate success, global_id 4432 2024-03-25T18:14:17.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f682984a640 10 monclient: _renew_subs 2024-03-25T18:14:17.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f682984a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:14:17.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f682984a640 1 -- 172.21.15.64:0/2393381269 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68241372b0 con 0x7f682414be90 2024-03-25T18:14:17.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f682984a640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:17.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f682984a640 10 monclient: _renew_subs 2024-03-25T18:14:17.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f682984a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:14:17.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f682984a640 1 -- 172.21.15.64:0/2393381269 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68241369c0 con 0x7f682414be90 2024-03-25T18:14:17.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.518+0000 7f682984a640 1 librados: init done 2024-03-25T18:14:17.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.519+0000 7f682984a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:17.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.519+0000 7f682984a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:17.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.519+0000 7f682984a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:14:17.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.519+0000 7f682984a640 1 -- 172.21.15.64:0/2393381269 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6824136f20 con 0x7f682414be90 2024-03-25T18:14:17.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.815+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f681c003990 con 0x7f682414be90 2024-03-25T18:14:17.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.815+0000 7f6820ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:17.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.815+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f681c003d20 con 0x7f682414be90 2024-03-25T18:14:17.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.815+0000 7f6820ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:17.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.815+0000 7f6820ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:17.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.815+0000 7f6820ff9640 10 monclient: dump: 2024-03-25T18:14:17.817 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:17.817 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:17.817 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:17.818 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:17.818 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:17.818 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:17.818 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:17.818 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:17.818 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:17.818 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:17.818 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.815+0000 7f6821ffb640 4 set_mon_vals no callback set 2024-03-25T18:14:17.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.816+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f681c003f40 con 0x7f682414be90 2024-03-25T18:14:17.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.817+0000 7f6822ffd640 10 monclient: get_auth_request con 0x7f680406de00 auth_method 0 2024-03-25T18:14:17.819 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.817+0000 7f6820ff9640 10 client.4432.objecter ms_handle_connect 0x7f680406de00 2024-03-25T18:14:17.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.827+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f681c054970 con 0x7f682414be90 2024-03-25T18:14:17.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.827+0000 7f6820ff9640 10 client.4432.objecter ms_dispatch 0x7f6824000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:17.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.827+0000 7f6820ff9640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:17.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.827+0000 7f6820ff9640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:17.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.827+0000 7f6820ff9640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-25T18:14:17.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.831+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f681c050480 con 0x7f682414be90 2024-03-25T18:14:17.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.831+0000 7f6820ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:17.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:17.831+0000 7f6820ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:18.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.086+0000 7f682984a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:18.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.086+0000 7f682984a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:18.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.086+0000 7f682984a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:14:18.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.086+0000 7f682984a640 1 -- 172.21.15.64:0/2393381269 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68241441a0 con 0x7f682414be90 2024-03-25T18:14:18.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.122+0000 7f6820ff9640 1 -- 172.21.15.64:0/2393381269 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f681c0523b0 con 0x7f682414be90 2024-03-25T18:14:18.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.122+0000 7f6820ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:18.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.122+0000 7f6820ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:18.124 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:14:18.125 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:18.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.125+0000 7f682984a640 10 librados: watch_flush enter 2024-03-25T18:14:18.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.125+0000 7f682984a640 10 librados: watch_flush exit 2024-03-25T18:14:18.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.125+0000 7f682984a640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-25T18:14:18.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.125+0000 7f682984a640 10 client.4432.objecter successfully canceled tick 2024-03-25T18:14:18.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.125+0000 7f682984a640 1 -- 172.21.15.64:0/2393381269 >> v1:172.21.15.64:6800/2915543638 conn(0x7f680406de00 legacy=0x7f68040702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:18.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.125+0000 7f682984a640 10 monclient: shutdown 2024-03-25T18:14:18.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.125+0000 7f682984a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:18.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.125+0000 7f682984a640 1 -- 172.21.15.64:0/2393381269 >> v1:172.21.15.86:6789/0 conn(0x7f682414be90 legacy=0x7f68241427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:18.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.125+0000 7f682984a640 1 -- 172.21.15.64:0/2393381269 shutdown_connections 2024-03-25T18:14:18.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.125+0000 7f682984a640 1 -- 172.21.15.64:0/2393381269 >> 172.21.15.64:0/2393381269 conn(0x7f68240617a0 msgr2=0x7f68240b2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:18.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.125+0000 7f682984a640 1 -- 172.21.15.64:0/2393381269 shutdown_connections 2024-03-25T18:14:18.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.125+0000 7f682984a640 1 -- 172.21.15.64:0/2393381269 wait complete. 2024-03-25T18:14:18.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:18.125+0000 7f682984a640 1 librados: shutdown 2024-03-25T18:14:18.138 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:24.945 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:25.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.060+0000 7efd5de8d640 10 monclient: get_monmap_and_config 2024-03-25T18:14:25.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.060+0000 7efd5de8d640 10 monclient: build_initial_monmap 2024-03-25T18:14:25.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.060+0000 7efd5de8d640 10 monclient: monmap: 2024-03-25T18:14:25.062 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:14:25.062 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:25.062 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:14:25.061774+0000 2024-03-25T18:14:25.062 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:14:25.061774+0000 2024-03-25T18:14:25.062 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:25.062 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:25.062 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:14:25.062 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:14:25.062 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:14:25.062 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.061+0000 7efd5de8d640 1 Processor -- start 2024-03-25T18:14:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.061+0000 7efd5de8d640 1 -- start start 2024-03-25T18:14:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.061+0000 7efd5de8d640 10 monclient: init 2024-03-25T18:14:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.062+0000 7efd5de8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.062+0000 7efd5de8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:25.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.062+0000 7efd5de8d640 10 monclient(hunting): picked mon.noname-a con 0x7efd580a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:14:25.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.062+0000 7efd5de8d640 10 monclient(hunting): picked mon.noname-c con 0x7efd5813f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:14:25.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.062+0000 7efd5de8d640 10 monclient(hunting): picked mon.noname-b con 0x7efd581430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:14:25.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.062+0000 7efd5de8d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd58142520 con 0x7efd580a98f0 2024-03-25T18:14:25.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.062+0000 7efd5de8d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd580b28a0 con 0x7efd5813f710 2024-03-25T18:14:25.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.062+0000 7efd5de8d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd580b2a30 con 0x7efd581430c0 2024-03-25T18:14:25.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.062+0000 7efd5de8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:25.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.062+0000 7efd5de8d640 10 monclient(hunting): authenticate will time out at 7251.300996s 2024-03-25T18:14:25.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.068+0000 7efd56ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7efd5813f710 0x7efd581429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:45012/0 (socket says 172.21.15.64:45012) 2024-03-25T18:14:25.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.068+0000 7efd56ffd640 1 -- 172.21.15.64:0/1098178274 learned_addr learned my addr 172.21.15.64:0/1098178274 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:14:25.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.089+0000 7efd567fc640 1 -- 172.21.15.64:0/1098178274 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067042684 0 0) 0x7efd580b2a30 con 0x7efd581430c0 2024-03-25T18:14:25.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.090+0000 7efd567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:25.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.090+0000 7efd567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:25.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.090+0000 7efd567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:25.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.090+0000 7efd567fc640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:14:25.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.090+0000 7efd567fc640 1 -- 172.21.15.64:0/1098178274 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd40003520 con 0x7efd581430c0 2024-03-25T18:14:25.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.102+0000 7efd567fc640 1 -- 172.21.15.64:0/1098178274 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467040300 0 0) 0x7efd580b28a0 con 0x7efd5813f710 2024-03-25T18:14:25.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.102+0000 7efd567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:25.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.102+0000 7efd567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:25.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.102+0000 7efd567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:25.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.102+0000 7efd567fc640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:14:25.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.103+0000 7efd567fc640 1 -- 172.21.15.64:0/1098178274 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd580b2a30 con 0x7efd5813f710 2024-03-25T18:14:25.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.109+0000 7efd567fc640 1 -- 172.21.15.64:0/1098178274 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783022554 0 0) 0x7efd58142520 con 0x7efd580a98f0 2024-03-25T18:14:25.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.109+0000 7efd567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:25.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.109+0000 7efd567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:25.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.109+0000 7efd567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:25.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.109+0000 7efd567fc640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:14:25.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.109+0000 7efd567fc640 1 -- 172.21.15.64:0/1098178274 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd580b28a0 con 0x7efd580a98f0 2024-03-25T18:14:25.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.112+0000 7efd567fc640 1 -- 172.21.15.64:0/1098178274 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2432347008 0 0) 0x7efd40003520 con 0x7efd581430c0 2024-03-25T18:14:25.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.112+0000 7efd567fc640 1 -- 172.21.15.64:0/1098178274 >> v1:172.21.15.86:6789/0 conn(0x7efd5813f710 legacy=0x7efd581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:25.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.112+0000 7efd567fc640 1 -- 172.21.15.64:0/1098178274 >> v1:172.21.15.64:6789/0 conn(0x7efd580a98f0 legacy=0x7efd580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:25.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.112+0000 7efd567fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:25.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.112+0000 7efd567fc640 1 monclient: found mon.noname-b 2024-03-25T18:14:25.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.112+0000 7efd567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:25.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.112+0000 7efd567fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:14:25.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.112+0000 7efd567fc640 1 -- 172.21.15.64:0/1098178274 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd580b44c0 con 0x7efd581430c0 2024-03-25T18:14:25.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.112+0000 7efd567fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:25.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.112+0000 7efd567fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:25.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.112+0000 7efd567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:25.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.112+0000 7efd567fc640 1 -- 172.21.15.64:0/1098178274 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7efd50001a10 con 0x7efd581430c0 2024-03-25T18:14:25.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.112+0000 7efd567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:25.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.112+0000 7efd567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:25.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.112+0000 7efd567fc640 10 monclient: dump: 2024-03-25T18:14:25.114 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:25.115 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:25.115 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:25.115 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:25.115 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:25.115 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:25.115 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:25.115 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:25.115 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:25.115 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:25.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.112+0000 7efd5de8d640 5 monclient: authenticate success, global_id 4442 2024-03-25T18:14:25.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.112+0000 7efd5de8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:25.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.163+0000 7efd567fc640 1 -- 172.21.15.64:0/1098178274 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd50001b70 con 0x7efd581430c0 2024-03-25T18:14:25.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.163+0000 7efd567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:25.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.163+0000 7efd567fc640 1 -- 172.21.15.64:0/1098178274 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7efd50001f80 con 0x7efd581430c0 2024-03-25T18:14:25.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.163+0000 7efd5de8d640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:25.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.163+0000 7efd5de8d640 4 set_mon_vals no callback set 2024-03-25T18:14:25.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.163+0000 7efd567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:25.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.163+0000 7efd567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:25.164 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.163+0000 7efd567fc640 10 monclient: dump: 2024-03-25T18:14:25.165 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:25.165 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:25.165 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:25.165 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:25.165 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:25.165 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:25.165 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:25.165 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:25.165 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:25.165 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:25.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.164+0000 7efd5de8d640 10 monclient: shutdown 2024-03-25T18:14:25.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.164+0000 7efd5de8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:25.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.164+0000 7efd5de8d640 1 -- 172.21.15.64:0/1098178274 >> v1:172.21.15.64:6790/0 conn(0x7efd581430c0 legacy=0x7efd5814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:25.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.164+0000 7efd5de8d640 1 -- 172.21.15.64:0/1098178274 shutdown_connections 2024-03-25T18:14:25.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.164+0000 7efd5de8d640 1 -- 172.21.15.64:0/1098178274 >> 172.21.15.64:0/1098178274 conn(0x7efd580b7ea0 msgr2=0x7efd580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:25.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.164+0000 7efd5de8d640 1 -- 172.21.15.64:0/1098178274 shutdown_connections 2024-03-25T18:14:25.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.164+0000 7efd5de8d640 1 -- 172.21.15.64:0/1098178274 wait complete. 2024-03-25T18:14:25.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.165+0000 7efd5de8d640 10 monclient: build_initial_monmap 2024-03-25T18:14:25.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.165+0000 7efd5de8d640 10 monclient: monmap: 2024-03-25T18:14:25.167 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:14:25.167 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:25.167 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:14:25.167 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:14:25.167 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:25.167 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:14:25.168 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:14:25.168 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:14:25.168 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:14:25.168 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:25.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.165+0000 7efd5de8d640 1 librados: starting msgr at 2024-03-25T18:14:25.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.165+0000 7efd5de8d640 1 librados: starting objecter 2024-03-25T18:14:25.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.165+0000 7efd5de8d640 1 Processor -- start 2024-03-25T18:14:25.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.165+0000 7efd5de8d640 1 -- start start 2024-03-25T18:14:25.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.166+0000 7efd5de8d640 1 librados: setting wanted keys 2024-03-25T18:14:25.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.166+0000 7efd5de8d640 1 librados: calling monclient init 2024-03-25T18:14:25.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.166+0000 7efd5de8d640 10 monclient: init 2024-03-25T18:14:25.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.166+0000 7efd5de8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:25.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.166+0000 7efd5de8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:25.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.166+0000 7efd5de8d640 10 monclient(hunting): picked mon.noname-c con 0x7efd580a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:14:25.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.166+0000 7efd5de8d640 10 monclient(hunting): picked mon.noname-a con 0x7efd5813f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:14:25.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.166+0000 7efd5de8d640 10 monclient(hunting): picked mon.noname-b con 0x7efd581430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:14:25.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.166+0000 7efd5de8d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5813c2b0 con 0x7efd5813f710 2024-03-25T18:14:25.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.166+0000 7efd5de8d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd58136f10 con 0x7efd581430c0 2024-03-25T18:14:25.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.167+0000 7efd5de8d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd581370a0 con 0x7efd580a98f0 2024-03-25T18:14:25.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.167+0000 7efd5de8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:25.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.167+0000 7efd5de8d640 10 monclient(hunting): authenticate will time out at 7251.405493s 2024-03-25T18:14:25.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.172+0000 7efd577fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7efd580a98f0 0x7efd581d7590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:38446/0 (socket says 172.21.15.64:38446) 2024-03-25T18:14:25.173 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.173+0000 7efd577fe640 1 -- 172.21.15.64:0/1665006079 learned_addr learned my addr 172.21.15.64:0/1665006079 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:14:25.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.174+0000 7efd3ffff640 10 client.?.objecter ms_handle_connect 0x7efd580a98f0 2024-03-25T18:14:25.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.174+0000 7efd3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:25.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.177+0000 7efd3ffff640 10 client.?.objecter ms_handle_connect 0x7efd581430c0 2024-03-25T18:14:25.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.177+0000 7efd3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:25.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.180+0000 7efd3ffff640 1 -- 172.21.15.64:0/1665006079 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388905459 0 0) 0x7efd581370a0 con 0x7efd580a98f0 2024-03-25T18:14:25.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.180+0000 7efd3ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:25.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.180+0000 7efd3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:25.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.180+0000 7efd3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:25.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.180+0000 7efd3ffff640 10 monclient(hunting): my global_id is 4445 2024-03-25T18:14:25.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.180+0000 7efd3ffff640 1 -- 172.21.15.64:0/1665006079 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd44004eb0 con 0x7efd580a98f0 2024-03-25T18:14:25.184 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.183+0000 7efd3ffff640 10 client.?.objecter ms_handle_connect 0x7efd5813f710 2024-03-25T18:14:25.184 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.183+0000 7efd3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:25.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.189+0000 7efd3ffff640 1 -- 172.21.15.64:0/1665006079 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009916398 0 0) 0x7efd58136f10 con 0x7efd581430c0 2024-03-25T18:14:25.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.189+0000 7efd3ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:25.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.189+0000 7efd3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:25.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.189+0000 7efd3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:25.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.189+0000 7efd3ffff640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:14:25.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.189+0000 7efd3ffff640 1 -- 172.21.15.64:0/1665006079 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd581370a0 con 0x7efd581430c0 2024-03-25T18:14:25.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.191+0000 7efd3ffff640 1 -- 172.21.15.64:0/1665006079 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2687444576 0 0) 0x7efd44004eb0 con 0x7efd580a98f0 2024-03-25T18:14:25.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.191+0000 7efd3ffff640 1 -- 172.21.15.64:0/1665006079 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd58136f10 con 0x7efd580a98f0 2024-03-25T18:14:25.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.191+0000 7efd3ffff640 1 -- 172.21.15.64:0/1665006079 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7efd48001a10 con 0x7efd580a98f0 2024-03-25T18:14:25.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.191+0000 7efd3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:25.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.191+0000 7efd3ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:25.192 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.191+0000 7efd3ffff640 10 monclient(hunting): dump: 2024-03-25T18:14:25.192 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:25.192 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:25.192 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:25.192 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:25.193 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:25.193 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:25.193 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:25.193 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:25.193 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:25.193 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:25.193 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.191+0000 7efd3ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:25.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd3ffff640 1 -- 172.21.15.64:0/1665006079 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2166061703 0 0) 0x7efd58136f10 con 0x7efd580a98f0 2024-03-25T18:14:25.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd3ffff640 1 -- 172.21.15.64:0/1665006079 >> v1:172.21.15.86:6789/0 conn(0x7efd581430c0 legacy=0x7efd58133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:25.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd3ffff640 1 -- 172.21.15.64:0/1665006079 >> v1:172.21.15.64:6789/0 conn(0x7efd5813f710 legacy=0x7efd581377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:25.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd3ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:25.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd3ffff640 1 monclient: found mon.c 2024-03-25T18:14:25.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:25.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd3ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:14:25.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd3ffff640 1 -- 172.21.15.64:0/1665006079 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd58137230 con 0x7efd580a98f0 2024-03-25T18:14:25.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd3ffff640 10 monclient: _finish_auth 0 2024-03-25T18:14:25.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd3ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:25.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:25.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd5de8d640 5 monclient: authenticate success, global_id 4445 2024-03-25T18:14:25.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd5de8d640 10 monclient: _renew_subs 2024-03-25T18:14:25.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd5de8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:14:25.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd5de8d640 1 -- 172.21.15.64:0/1665006079 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5813b620 con 0x7efd580a98f0 2024-03-25T18:14:25.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd5de8d640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:25.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd5de8d640 10 monclient: _renew_subs 2024-03-25T18:14:25.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd5de8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:14:25.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd5de8d640 1 -- 172.21.15.64:0/1665006079 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd581338f0 con 0x7efd580a98f0 2024-03-25T18:14:25.201 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.199+0000 7efd5de8d640 1 librados: init done 2024-03-25T18:14:25.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.200+0000 7efd5de8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:25.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.200+0000 7efd5de8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:25.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.200+0000 7efd5de8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:14:25.202 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.200+0000 7efd5de8d640 1 -- 172.21.15.64:0/1665006079 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd5813b9c0 con 0x7efd580a98f0 2024-03-25T18:14:25.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.215+0000 7efd3ffff640 1 -- 172.21.15.64:0/1665006079 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd48001c70 con 0x7efd580a98f0 2024-03-25T18:14:25.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.215+0000 7efd3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:25.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.215+0000 7efd55ffb640 4 set_mon_vals no callback set 2024-03-25T18:14:25.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.532+0000 7efd3ffff640 1 -- 172.21.15.64:0/1665006079 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7efd480038c0 con 0x7efd580a98f0 2024-03-25T18:14:25.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.533+0000 7efd3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:25.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.533+0000 7efd3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:25.534 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.533+0000 7efd3ffff640 10 monclient: dump: 2024-03-25T18:14:25.534 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:25.534 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:25.534 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:25.534 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:25.534 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:25.534 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:25.534 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:25.534 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:25.534 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:25.534 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:25.535 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.533+0000 7efd3ffff640 1 -- 172.21.15.64:0/1665006079 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7efd48003aa0 con 0x7efd580a98f0 2024-03-25T18:14:25.535 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.534+0000 7efd56ffd640 10 monclient: get_auth_request con 0x7efd4406dd10 auth_method 0 2024-03-25T18:14:25.535 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.534+0000 7efd3ffff640 10 client.4445.objecter ms_handle_connect 0x7efd4406dd10 2024-03-25T18:14:25.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.541+0000 7efd3ffff640 1 -- 172.21.15.64:0/1665006079 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7efd48054750 con 0x7efd580a98f0 2024-03-25T18:14:25.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.541+0000 7efd3ffff640 10 client.4445.objecter ms_dispatch 0x7efd58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:25.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.541+0000 7efd3ffff640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:25.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.541+0000 7efd3ffff640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:25.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.541+0000 7efd3ffff640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-25T18:14:25.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.541+0000 7efd3ffff640 1 -- 172.21.15.64:0/1665006079 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd48050260 con 0x7efd580a98f0 2024-03-25T18:14:25.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.541+0000 7efd3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:25.542 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.541+0000 7efd3ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:25.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.777+0000 7efd5de8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:25.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.777+0000 7efd5de8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:25.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.777+0000 7efd5de8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:14:25.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.777+0000 7efd5de8d640 1 -- 172.21.15.64:0/1665006079 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd580627b0 con 0x7efd580a98f0 2024-03-25T18:14:25.824 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.824+0000 7efd3ffff640 1 -- 172.21.15.64:0/1665006079 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7efd48052190 con 0x7efd580a98f0 2024-03-25T18:14:25.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.824+0000 7efd3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:25.825 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.824+0000 7efd3ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:25.825 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:14:25.825 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:25.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.826+0000 7efd5de8d640 10 librados: watch_flush enter 2024-03-25T18:14:25.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.826+0000 7efd5de8d640 10 librados: watch_flush exit 2024-03-25T18:14:25.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.826+0000 7efd5de8d640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-25T18:14:25.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.826+0000 7efd5de8d640 10 client.4445.objecter successfully canceled tick 2024-03-25T18:14:25.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.826+0000 7efd5de8d640 1 -- 172.21.15.64:0/1665006079 >> v1:172.21.15.64:6800/2915543638 conn(0x7efd4406dd10 legacy=0x7efd440701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:25.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.826+0000 7efd5de8d640 10 monclient: shutdown 2024-03-25T18:14:25.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.826+0000 7efd5de8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:25.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.826+0000 7efd5de8d640 1 -- 172.21.15.64:0/1665006079 >> v1:172.21.15.64:6790/0 conn(0x7efd580a98f0 legacy=0x7efd581d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:25.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.826+0000 7efd5de8d640 1 -- 172.21.15.64:0/1665006079 shutdown_connections 2024-03-25T18:14:25.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.826+0000 7efd5de8d640 1 -- 172.21.15.64:0/1665006079 >> 172.21.15.64:0/1665006079 conn(0x7efd580b7ea0 msgr2=0x7efd58074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:25.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.827+0000 7efd5de8d640 1 -- 172.21.15.64:0/1665006079 shutdown_connections 2024-03-25T18:14:25.828 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.827+0000 7efd5de8d640 1 -- 172.21.15.64:0/1665006079 wait complete. 2024-03-25T18:14:25.829 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:25.827+0000 7efd5de8d640 1 librados: shutdown 2024-03-25T18:14:25.841 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:32.648 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:32.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.760+0000 7fb8d0063640 10 monclient: get_monmap_and_config 2024-03-25T18:14:32.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.760+0000 7fb8d0063640 10 monclient: build_initial_monmap 2024-03-25T18:14:32.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.760+0000 7fb8d0063640 10 monclient: monmap: 2024-03-25T18:14:32.761 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:14:32.761 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:32.761 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:14:32.761399+0000 2024-03-25T18:14:32.761 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:14:32.761399+0000 2024-03-25T18:14:32.761 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:32.761 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:32.762 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:14:32.762 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:14:32.762 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:14:32.762 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:32.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.761+0000 7fb8d0063640 1 Processor -- start 2024-03-25T18:14:32.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.761+0000 7fb8d0063640 1 -- start start 2024-03-25T18:14:32.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.761+0000 7fb8d0063640 10 monclient: init 2024-03-25T18:14:32.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.762+0000 7fb8d0063640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:32.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.762+0000 7fb8d0063640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:32.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.762+0000 7fb8d0063640 10 monclient(hunting): picked mon.noname-a con 0x7fb8c80a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:14:32.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.762+0000 7fb8d0063640 10 monclient(hunting): picked mon.noname-c con 0x7fb8c813f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:14:32.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.762+0000 7fb8d0063640 10 monclient(hunting): picked mon.noname-b con 0x7fb8c81430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:14:32.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.762+0000 7fb8d0063640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c8142520 con 0x7fb8c80a98f0 2024-03-25T18:14:32.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.762+0000 7fb8d0063640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c80b28a0 con 0x7fb8c813f710 2024-03-25T18:14:32.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.762+0000 7fb8d0063640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c80b2a30 con 0x7fb8c81430c0 2024-03-25T18:14:32.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.762+0000 7fb8d0063640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:32.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.762+0000 7fb8d0063640 10 monclient(hunting): authenticate will time out at 7259.001019s 2024-03-25T18:14:32.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.766+0000 7fb8ce600640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fb8c81430c0 0x7fb8c814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:33054/0 (socket says 172.21.15.64:33054) 2024-03-25T18:14:32.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.766+0000 7fb8ce600640 1 -- 172.21.15.64:0/4115698455 learned_addr learned my addr 172.21.15.64:0/4115698455 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:14:32.776 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.775+0000 7fb8ccdfd640 1 -- 172.21.15.64:0/4115698455 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199467433 0 0) 0x7fb8c80b2a30 con 0x7fb8c81430c0 2024-03-25T18:14:32.776 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.775+0000 7fb8ccdfd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:32.776 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.775+0000 7fb8ccdfd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:32.776 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.775+0000 7fb8ccdfd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:32.776 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.775+0000 7fb8ccdfd640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:14:32.776 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.775+0000 7fb8ccdfd640 1 -- 172.21.15.64:0/4115698455 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b4003520 con 0x7fb8c81430c0 2024-03-25T18:14:32.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.782+0000 7fb8ccdfd640 1 -- 172.21.15.64:0/4115698455 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3452847549 0 0) 0x7fb8b4003520 con 0x7fb8c81430c0 2024-03-25T18:14:32.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.782+0000 7fb8ccdfd640 1 -- 172.21.15.64:0/4115698455 >> v1:172.21.15.86:6789/0 conn(0x7fb8c813f710 legacy=0x7fb8c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:32.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.782+0000 7fb8ccdfd640 1 -- 172.21.15.64:0/4115698455 >> v1:172.21.15.64:6789/0 conn(0x7fb8c80a98f0 legacy=0x7fb8c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:32.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.782+0000 7fb8ccdfd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:32.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.782+0000 7fb8ccdfd640 1 monclient: found mon.noname-b 2024-03-25T18:14:32.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.782+0000 7fb8ccdfd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:32.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.782+0000 7fb8ccdfd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:14:32.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.782+0000 7fb8ccdfd640 1 -- 172.21.15.64:0/4115698455 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c80b44c0 con 0x7fb8c81430c0 2024-03-25T18:14:32.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.782+0000 7fb8ccdfd640 10 monclient: _finish_auth 0 2024-03-25T18:14:32.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.782+0000 7fb8ccdfd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:32.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.782+0000 7fb8ccdfd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:32.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.782+0000 7fb8d0063640 5 monclient: authenticate success, global_id 4448 2024-03-25T18:14:32.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.782+0000 7fb8d0063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:32.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.784+0000 7fb8ccdfd640 1 -- 172.21.15.64:0/4115698455 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb8c0001a10 con 0x7fb8c81430c0 2024-03-25T18:14:32.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.784+0000 7fb8ccdfd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:32.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.784+0000 7fb8ccdfd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:32.785 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.784+0000 7fb8ccdfd640 10 monclient: dump: 2024-03-25T18:14:32.785 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:32.785 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:32.785 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:32.785 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:32.785 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:32.785 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:32.785 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:32.785 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:32.785 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:32.785 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:32.786 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.784+0000 7fb8d0063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:32.794 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.793+0000 7fb8ccdfd640 1 -- 172.21.15.64:0/4115698455 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8c0001ca0 con 0x7fb8c81430c0 2024-03-25T18:14:32.794 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.793+0000 7fb8ccdfd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:32.794 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.793+0000 7fb8ccdfd640 1 -- 172.21.15.64:0/4115698455 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb8c0001fb0 con 0x7fb8c81430c0 2024-03-25T18:14:32.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.793+0000 7fb8ccdfd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:32.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.793+0000 7fb8ccdfd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:32.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.793+0000 7fb8ccdfd640 10 monclient: dump: 2024-03-25T18:14:32.795 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:32.795 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:32.795 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:32.795 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:32.795 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:32.795 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:32.795 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:32.795 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:32.795 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:32.795 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:32.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.793+0000 7fb8d0063640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:32.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.793+0000 7fb8d0063640 4 set_mon_vals no callback set 2024-03-25T18:14:32.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.794+0000 7fb8d0063640 10 monclient: shutdown 2024-03-25T18:14:32.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.794+0000 7fb8d0063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:32.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.794+0000 7fb8d0063640 1 -- 172.21.15.64:0/4115698455 >> v1:172.21.15.64:6790/0 conn(0x7fb8c81430c0 legacy=0x7fb8c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:32.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.794+0000 7fb8d0063640 1 -- 172.21.15.64:0/4115698455 shutdown_connections 2024-03-25T18:14:32.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.794+0000 7fb8d0063640 1 -- 172.21.15.64:0/4115698455 >> 172.21.15.64:0/4115698455 conn(0x7fb8c80b7ea0 msgr2=0x7fb8c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:32.798 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.794+0000 7fb8d0063640 1 -- 172.21.15.64:0/4115698455 shutdown_connections 2024-03-25T18:14:32.798 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.794+0000 7fb8d0063640 1 -- 172.21.15.64:0/4115698455 wait complete. 2024-03-25T18:14:32.798 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.795+0000 7fb8d0063640 10 monclient: build_initial_monmap 2024-03-25T18:14:32.798 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.795+0000 7fb8d0063640 10 monclient: monmap: 2024-03-25T18:14:32.798 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:14:32.798 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:32.798 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:14:32.798 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:14:32.798 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:32.798 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:14:32.798 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:14:32.798 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:14:32.798 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:14:32.799 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:32.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.795+0000 7fb8d0063640 1 librados: starting msgr at 2024-03-25T18:14:32.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.795+0000 7fb8d0063640 1 librados: starting objecter 2024-03-25T18:14:32.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.796+0000 7fb8d0063640 1 Processor -- start 2024-03-25T18:14:32.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.796+0000 7fb8d0063640 1 -- start start 2024-03-25T18:14:32.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.796+0000 7fb8d0063640 1 librados: setting wanted keys 2024-03-25T18:14:32.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.796+0000 7fb8d0063640 1 librados: calling monclient init 2024-03-25T18:14:32.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.796+0000 7fb8d0063640 10 monclient: init 2024-03-25T18:14:32.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.796+0000 7fb8d0063640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:32.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.796+0000 7fb8d0063640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:32.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.796+0000 7fb8d0063640 10 monclient(hunting): picked mon.noname-b con 0x7fb8c80a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:14:32.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.796+0000 7fb8d0063640 10 monclient(hunting): picked mon.noname-c con 0x7fb8c813f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:14:32.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.796+0000 7fb8d0063640 10 monclient(hunting): picked mon.noname-a con 0x7fb8c81430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:14:32.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.796+0000 7fb8d0063640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c81d7500 con 0x7fb8c81430c0 2024-03-25T18:14:32.799 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.796+0000 7fb8d0063640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c81d8380 con 0x7fb8c80a98f0 2024-03-25T18:14:32.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.796+0000 7fb8d0063640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c80aba90 con 0x7fb8c813f710 2024-03-25T18:14:32.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.796+0000 7fb8d0063640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:32.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.796+0000 7fb8d0063640 10 monclient(hunting): authenticate will time out at 7259.035267s 2024-03-25T18:14:32.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.798+0000 7fb8ce600640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fb8c81430c0 0x7fb8c8133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:59392/0 (socket says 172.21.15.64:59392) 2024-03-25T18:14:32.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.798+0000 7fb8ce600640 1 -- 172.21.15.64:0/2215007952 learned_addr learned my addr 172.21.15.64:0/2215007952 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:14:32.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.798+0000 7fb8be7fc640 10 client.?.objecter ms_handle_connect 0x7fb8c81430c0 2024-03-25T18:14:32.801 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.798+0000 7fb8be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:32.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.802+0000 7fb8be7fc640 10 client.?.objecter ms_handle_connect 0x7fb8c80a98f0 2024-03-25T18:14:32.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.802+0000 7fb8be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:32.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.877+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375730543 0 0) 0x7fb8c81d7500 con 0x7fb8c81430c0 2024-03-25T18:14:32.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.877+0000 7fb8be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:32.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.877+0000 7fb8be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:32.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.877+0000 7fb8be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:32.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.877+0000 7fb8be7fc640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:14:32.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.877+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8ac004eb0 con 0x7fb8c81430c0 2024-03-25T18:14:32.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.885+0000 7fb8be7fc640 10 client.?.objecter ms_handle_connect 0x7fb8c813f710 2024-03-25T18:14:32.886 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.885+0000 7fb8be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:32.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.894+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747453950 0 0) 0x7fb8c80aba90 con 0x7fb8c813f710 2024-03-25T18:14:32.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.894+0000 7fb8be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:32.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.894+0000 7fb8be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:32.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.894+0000 7fb8be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:32.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.894+0000 7fb8be7fc640 10 monclient(hunting): my global_id is 4451 2024-03-25T18:14:32.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.894+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8c81d7500 con 0x7fb8c813f710 2024-03-25T18:14:32.907 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.906+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3868431877 0 0) 0x7fb8c81d7500 con 0x7fb8c813f710 2024-03-25T18:14:32.907 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.906+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8c80aba90 con 0x7fb8c813f710 2024-03-25T18:14:32.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.906+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb8b80018b0 con 0x7fb8c813f710 2024-03-25T18:14:32.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.906+0000 7fb8be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:32.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.906+0000 7fb8be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:32.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.906+0000 7fb8be7fc640 10 monclient(hunting): dump: 2024-03-25T18:14:32.908 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:32.908 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:32.908 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:32.908 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:32.908 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:32.908 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:32.908 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:32.908 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:32.908 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:32.908 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:32.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.906+0000 7fb8be7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:32.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.910+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4231157691 0 0) 0x7fb8ac004eb0 con 0x7fb8c81430c0 2024-03-25T18:14:32.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.910+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8c81d7500 con 0x7fb8c81430c0 2024-03-25T18:14:32.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.910+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb8c00019d0 con 0x7fb8c81430c0 2024-03-25T18:14:32.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.910+0000 7fb8be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:32.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.910+0000 7fb8be7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:32.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.910+0000 7fb8be7fc640 10 monclient(hunting): dump: 2024-03-25T18:14:32.912 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:32.912 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:32.912 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:32.912 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:32.912 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:32.912 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:32.912 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:32.912 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:32.912 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:32.912 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:32.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.914+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3423442556 0 0) 0x7fb8c80aba90 con 0x7fb8c813f710 2024-03-25T18:14:32.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.914+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 >> v1:172.21.15.86:6789/0 conn(0x7fb8c80a98f0 legacy=0x7fb8c8137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:32.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.914+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 >> v1:172.21.15.64:6789/0 conn(0x7fb8c81430c0 legacy=0x7fb8c8133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:32.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.914+0000 7fb8be7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:32.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.914+0000 7fb8be7fc640 1 monclient: found mon.c 2024-03-25T18:14:32.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.914+0000 7fb8be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:32.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.914+0000 7fb8be7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:14:32.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.914+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c8134e70 con 0x7fb8c813f710 2024-03-25T18:14:32.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.914+0000 7fb8be7fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:32.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.914+0000 7fb8be7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:32.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.914+0000 7fb8be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:32.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.914+0000 7fb8d0063640 5 monclient: authenticate success, global_id 4451 2024-03-25T18:14:32.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.914+0000 7fb8d0063640 10 monclient: _renew_subs 2024-03-25T18:14:32.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.914+0000 7fb8d0063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:14:32.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.914+0000 7fb8d0063640 1 -- 172.21.15.64:0/2215007952 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c8135000 con 0x7fb8c813f710 2024-03-25T18:14:32.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.914+0000 7fb8d0063640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:32.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.914+0000 7fb8d0063640 10 monclient: _renew_subs 2024-03-25T18:14:32.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.915+0000 7fb8d0063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:14:32.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.915+0000 7fb8d0063640 1 -- 172.21.15.64:0/2215007952 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c81384b0 con 0x7fb8c813f710 2024-03-25T18:14:32.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.915+0000 7fb8d0063640 1 librados: init done 2024-03-25T18:14:32.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.915+0000 7fb8d0063640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:32.917 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.915+0000 7fb8d0063640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:32.918 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.915+0000 7fb8d0063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:14:32.918 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:32.915+0000 7fb8d0063640 1 -- 172.21.15.64:0/2215007952 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c81d7ac0 con 0x7fb8c813f710 2024-03-25T18:14:33.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.286+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8b8003610 con 0x7fb8c813f710 2024-03-25T18:14:33.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.286+0000 7fb8be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:33.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.286+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb8b8003a60 con 0x7fb8c813f710 2024-03-25T18:14:33.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.286+0000 7fb8be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:33.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.286+0000 7fb8be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:33.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.286+0000 7fb8be7fc640 10 monclient: dump: 2024-03-25T18:14:33.288 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:33.288 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:33.288 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:33.288 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:33.289 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:33.289 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:33.289 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:33.289 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:33.289 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:33.289 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:33.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.286+0000 7fb8bffff640 4 set_mon_vals no callback set 2024-03-25T18:14:33.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.287+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fb8b8003c80 con 0x7fb8c813f710 2024-03-25T18:14:33.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.288+0000 7fb8cddff640 10 monclient: get_auth_request con 0x7fb8ac06de90 auth_method 0 2024-03-25T18:14:33.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.288+0000 7fb8be7fc640 10 client.4451.objecter ms_handle_connect 0x7fb8ac06de90 2024-03-25T18:14:33.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.299+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fb8b8053550 con 0x7fb8c813f710 2024-03-25T18:14:33.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.299+0000 7fb8be7fc640 10 client.4451.objecter ms_dispatch 0x7fb8c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:33.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.299+0000 7fb8be7fc640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:33.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.299+0000 7fb8be7fc640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:33.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.299+0000 7fb8be7fc640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-25T18:14:33.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.299+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb8b804f060 con 0x7fb8c813f710 2024-03-25T18:14:33.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.299+0000 7fb8be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:33.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.299+0000 7fb8be7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:33.549 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.548+0000 7fb8d0063640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:33.549 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.548+0000 7fb8d0063640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:33.549 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.548+0000 7fb8d0063640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:14:33.549 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.548+0000 7fb8d0063640 1 -- 172.21.15.64:0/2215007952 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8c814f2d0 con 0x7fb8c813f710 2024-03-25T18:14:33.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.600+0000 7fb8be7fc640 1 -- 172.21.15.64:0/2215007952 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fb8b8050f90 con 0x7fb8c813f710 2024-03-25T18:14:33.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.600+0000 7fb8be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:33.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.600+0000 7fb8be7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:33.601 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:14:33.602 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:33.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.602+0000 7fb8d0063640 10 librados: watch_flush enter 2024-03-25T18:14:33.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.602+0000 7fb8d0063640 10 librados: watch_flush exit 2024-03-25T18:14:33.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.602+0000 7fb8d0063640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-25T18:14:33.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.602+0000 7fb8d0063640 10 client.4451.objecter successfully canceled tick 2024-03-25T18:14:33.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.602+0000 7fb8d0063640 1 -- 172.21.15.64:0/2215007952 >> v1:172.21.15.64:6800/2915543638 conn(0x7fb8ac06de90 legacy=0x7fb8ac070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:33.604 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.602+0000 7fb8d0063640 10 monclient: shutdown 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.602+0000 7fb8d0063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.602+0000 7fb8d0063640 1 -- 172.21.15.64:0/2215007952 >> v1:172.21.15.64:6790/0 conn(0x7fb8c813f710 legacy=0x7fb8c81d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.602+0000 7fb8d0063640 1 -- 172.21.15.64:0/2215007952 shutdown_connections 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.603+0000 7fb8d0063640 1 -- 172.21.15.64:0/2215007952 >> 172.21.15.64:0/2215007952 conn(0x7fb8c80b7ea0 msgr2=0x7fb8c8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.603+0000 7fb8d0063640 1 -- 172.21.15.64:0/2215007952 shutdown_connections 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.603+0000 7fb8d0063640 1 -- 172.21.15.64:0/2215007952 wait complete. 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:33.603+0000 7fb8d0063640 1 librados: shutdown 2024-03-25T18:14:33.614 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:37.353 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:05:20.465 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:37.358 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:05:20.470 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:37.394 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:05:20.507 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:37.407 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:05:20.519 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:37.417 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:05:20.496 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:37.421 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:05:20.533 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:37.421 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:05:20.500 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:37.447 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:05:20.560 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:37.448 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:05:20.527 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:40.420 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:40.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.535+0000 7f4520fc6640 10 monclient: get_monmap_and_config 2024-03-25T18:14:40.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.535+0000 7f4520fc6640 10 monclient: build_initial_monmap 2024-03-25T18:14:40.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.535+0000 7f4520fc6640 10 monclient: monmap: 2024-03-25T18:14:40.537 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:14:40.537 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:40.537 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:14:40.536769+0000 2024-03-25T18:14:40.537 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:14:40.536769+0000 2024-03-25T18:14:40.537 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:40.537 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:40.537 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:14:40.537 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:14:40.537 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:14:40.537 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:40.538 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.536+0000 7f4520fc6640 1 Processor -- start 2024-03-25T18:14:40.538 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.536+0000 7f4520fc6640 1 -- start start 2024-03-25T18:14:40.538 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.536+0000 7f4520fc6640 10 monclient: init 2024-03-25T18:14:40.538 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.537+0000 7f4520fc6640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:40.538 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.537+0000 7f4520fc6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:40.538 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.537+0000 7f4520fc6640 10 monclient(hunting): picked mon.noname-a con 0x7f451c0a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:14:40.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.537+0000 7f4520fc6640 10 monclient(hunting): picked mon.noname-b con 0x7f451c13f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:14:40.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.537+0000 7f4520fc6640 10 monclient(hunting): picked mon.noname-c con 0x7f451c1430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:14:40.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.537+0000 7f4520fc6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c142520 con 0x7f451c0a98f0 2024-03-25T18:14:40.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.537+0000 7f4520fc6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c0b28a0 con 0x7f451c1430c0 2024-03-25T18:14:40.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.537+0000 7f4520fc6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c0b2a30 con 0x7f451c13f710 2024-03-25T18:14:40.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.537+0000 7f4520fc6640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:40.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.537+0000 7f4520fc6640 10 monclient(hunting): authenticate will time out at 7266.775979s 2024-03-25T18:14:40.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.548+0000 7f4519d9b640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f451c13f710 0x7f451c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:52638/0 (socket says 172.21.15.64:52638) 2024-03-25T18:14:40.549 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.548+0000 7f4519d9b640 1 -- 172.21.15.64:0/560303100 learned_addr learned my addr 172.21.15.64:0/560303100 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:14:40.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.567+0000 7f451959a640 1 -- 172.21.15.64:0/560303100 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837907665 0 0) 0x7f451c0b2a30 con 0x7f451c13f710 2024-03-25T18:14:40.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.567+0000 7f451959a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.567+0000 7f451959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.567+0000 7f451959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.567+0000 7f451959a640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:14:40.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.567+0000 7f451959a640 1 -- 172.21.15.64:0/560303100 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4504003520 con 0x7f451c13f710 2024-03-25T18:14:40.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.568+0000 7f451959a640 1 -- 172.21.15.64:0/560303100 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501467817 0 0) 0x7f451c0b28a0 con 0x7f451c1430c0 2024-03-25T18:14:40.568 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.568+0000 7f451959a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.568+0000 7f451959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.568+0000 7f451959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.568+0000 7f451959a640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:14:40.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.568+0000 7f451959a640 1 -- 172.21.15.64:0/560303100 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f451c0b2a30 con 0x7f451c1430c0 2024-03-25T18:14:40.580 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.579+0000 7f451959a640 1 -- 172.21.15.64:0/560303100 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606644296 0 0) 0x7f451c142520 con 0x7f451c0a98f0 2024-03-25T18:14:40.580 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.579+0000 7f451959a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.580 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.579+0000 7f451959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.580 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.579+0000 7f451959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.580 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.579+0000 7f451959a640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:14:40.580 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.579+0000 7f451959a640 1 -- 172.21.15.64:0/560303100 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f451c0b28a0 con 0x7f451c0a98f0 2024-03-25T18:14:40.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.599+0000 7f451959a640 1 -- 172.21.15.64:0/560303100 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 593853809 0 0) 0x7f451c0b2a30 con 0x7f451c1430c0 2024-03-25T18:14:40.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.599+0000 7f451959a640 1 -- 172.21.15.64:0/560303100 >> v1:172.21.15.64:6790/0 conn(0x7f451c13f710 legacy=0x7f451c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.599+0000 7f451959a640 1 -- 172.21.15.64:0/560303100 >> v1:172.21.15.64:6789/0 conn(0x7f451c0a98f0 legacy=0x7f451c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.599+0000 7f451959a640 10 monclient: _finish_hunting 0 2024-03-25T18:14:40.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.599+0000 7f451959a640 1 monclient: found mon.noname-c 2024-03-25T18:14:40.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.599+0000 7f451959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:40.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.599+0000 7f451959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:14:40.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.599+0000 7f451959a640 1 -- 172.21.15.64:0/560303100 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451c0b44c0 con 0x7f451c1430c0 2024-03-25T18:14:40.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.599+0000 7f451959a640 10 monclient: _finish_auth 0 2024-03-25T18:14:40.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.599+0000 7f451959a640 10 monclient: _check_auth_tickets 2024-03-25T18:14:40.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.599+0000 7f451959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:40.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.599+0000 7f451959a640 1 -- 172.21.15.64:0/560303100 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f45140019f0 con 0x7f451c1430c0 2024-03-25T18:14:40.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.599+0000 7f4520fc6640 5 monclient: authenticate success, global_id 4447 2024-03-25T18:14:40.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.599+0000 7f4520fc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:40.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.599+0000 7f451959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:40.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.599+0000 7f451959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:40.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.599+0000 7f451959a640 10 monclient: dump: 2024-03-25T18:14:40.601 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:40.601 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:40.601 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:40.601 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:40.601 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:40.601 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:40.601 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:40.602 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:40.602 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:40.602 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:40.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.599+0000 7f4520fc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:40.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.608+0000 7f451959a640 1 -- 172.21.15.64:0/560303100 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4514001b50 con 0x7f451c1430c0 2024-03-25T18:14:40.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.608+0000 7f451959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:40.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.608+0000 7f451959a640 1 -- 172.21.15.64:0/560303100 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4514001f60 con 0x7f451c1430c0 2024-03-25T18:14:40.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.608+0000 7f451959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:40.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.608+0000 7f451959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:40.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.608+0000 7f451959a640 10 monclient: dump: 2024-03-25T18:14:40.609 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:40.609 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:40.609 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:40.609 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:40.609 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:40.609 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:40.609 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:40.609 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:40.610 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:40.610 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:40.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.608+0000 7f4520fc6640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:40.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.608+0000 7f4520fc6640 4 set_mon_vals no callback set 2024-03-25T18:14:40.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.609+0000 7f4520fc6640 10 monclient: shutdown 2024-03-25T18:14:40.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.609+0000 7f4520fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:40.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.609+0000 7f4520fc6640 1 -- 172.21.15.64:0/560303100 >> v1:172.21.15.86:6789/0 conn(0x7f451c1430c0 legacy=0x7f451c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.609+0000 7f4520fc6640 1 -- 172.21.15.64:0/560303100 shutdown_connections 2024-03-25T18:14:40.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.609+0000 7f4520fc6640 1 -- 172.21.15.64:0/560303100 >> 172.21.15.64:0/560303100 conn(0x7f451c0b7ea0 msgr2=0x7f451c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:40.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.609+0000 7f4520fc6640 1 -- 172.21.15.64:0/560303100 shutdown_connections 2024-03-25T18:14:40.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.609+0000 7f4520fc6640 1 -- 172.21.15.64:0/560303100 wait complete. 2024-03-25T18:14:40.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.610+0000 7f4520fc6640 10 monclient: build_initial_monmap 2024-03-25T18:14:40.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.610+0000 7f4520fc6640 10 monclient: monmap: 2024-03-25T18:14:40.612 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:14:40.612 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:40.612 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:14:40.612 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:14:40.612 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:40.612 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:14:40.612 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:14:40.612 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:14:40.612 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:14:40.612 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:40.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.610+0000 7f4520fc6640 1 librados: starting msgr at 2024-03-25T18:14:40.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.610+0000 7f4520fc6640 1 librados: starting objecter 2024-03-25T18:14:40.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.610+0000 7f4520fc6640 1 Processor -- start 2024-03-25T18:14:40.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.610+0000 7f4520fc6640 1 -- start start 2024-03-25T18:14:40.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.610+0000 7f4520fc6640 1 librados: setting wanted keys 2024-03-25T18:14:40.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.610+0000 7f4520fc6640 1 librados: calling monclient init 2024-03-25T18:14:40.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.610+0000 7f4520fc6640 10 monclient: init 2024-03-25T18:14:40.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.611+0000 7f4520fc6640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:40.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.611+0000 7f4520fc6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:40.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.611+0000 7f4520fc6640 10 monclient(hunting): picked mon.noname-b con 0x7f451c0a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:14:40.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.611+0000 7f4520fc6640 10 monclient(hunting): picked mon.noname-a con 0x7f451c13f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:14:40.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.611+0000 7f4520fc6640 10 monclient(hunting): picked mon.noname-c con 0x7f451c1430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:14:40.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.611+0000 7f4520fc6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c133ee0 con 0x7f451c13f710 2024-03-25T18:14:40.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.611+0000 7f4520fc6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c0abe30 con 0x7f451c0a98f0 2024-03-25T18:14:40.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.611+0000 7f4520fc6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c0ac0a0 con 0x7f451c1430c0 2024-03-25T18:14:40.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.611+0000 7f4520fc6640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:40.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.611+0000 7f4520fc6640 10 monclient(hunting): authenticate will time out at 7266.849982s 2024-03-25T18:14:40.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.617+0000 7f451ad9d640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f451c1430c0 0x7f451c138dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:52694/0 (socket says 172.21.15.64:52694) 2024-03-25T18:14:40.618 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.617+0000 7f451ad9d640 1 -- 172.21.15.64:0/4061246515 learned_addr learned my addr 172.21.15.64:0/4061246515 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:14:40.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.624+0000 7f450affd640 10 client.?.objecter ms_handle_connect 0x7f451c1430c0 2024-03-25T18:14:40.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.624+0000 7f450affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:40.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.637+0000 7f450affd640 10 client.?.objecter ms_handle_connect 0x7f451c13f710 2024-03-25T18:14:40.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.637+0000 7f450affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:40.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.637+0000 7f450affd640 10 client.?.objecter ms_handle_connect 0x7f451c0a98f0 2024-03-25T18:14:40.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.637+0000 7f450affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:40.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.638+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115144947 0 0) 0x7f451c0ac0a0 con 0x7f451c1430c0 2024-03-25T18:14:40.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.638+0000 7f450affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.638+0000 7f450affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.638+0000 7f450affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.638+0000 7f450affd640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:14:40.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.638+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44fc004eb0 con 0x7f451c1430c0 2024-03-25T18:14:40.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.643+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402767457 0 0) 0x7f451c0abe30 con 0x7f451c0a98f0 2024-03-25T18:14:40.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.643+0000 7f450affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.643+0000 7f450affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.643+0000 7f450affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.643+0000 7f450affd640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:14:40.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.643+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f451c0ac0a0 con 0x7f451c0a98f0 2024-03-25T18:14:40.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.662+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053328818 0 0) 0x7f4504003520 con 0x7f451c13f710 2024-03-25T18:14:40.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.662+0000 7f450affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.662+0000 7f450affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.662+0000 7f450affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.662+0000 7f450affd640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:14:40.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.662+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f451c0abe30 con 0x7f451c13f710 2024-03-25T18:14:40.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.663+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 531776943 0 0) 0x7f44fc004eb0 con 0x7f451c1430c0 2024-03-25T18:14:40.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.663+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4504003520 con 0x7f451c1430c0 2024-03-25T18:14:40.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.663+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4514001fa0 con 0x7f451c1430c0 2024-03-25T18:14:40.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.663+0000 7f450affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:40.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.663+0000 7f450affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:40.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.663+0000 7f450affd640 10 monclient(hunting): dump: 2024-03-25T18:14:40.664 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:40.664 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:40.664 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:40.664 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:40.664 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:40.664 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:40.664 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:40.664 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:40.664 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:40.664 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:40.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.663+0000 7f450affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:40.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.669+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 529389202 0 0) 0x7f451c0ac0a0 con 0x7f451c0a98f0 2024-03-25T18:14:40.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.669+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44fc004eb0 con 0x7f451c0a98f0 2024-03-25T18:14:40.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.669+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f450c001820 con 0x7f451c0a98f0 2024-03-25T18:14:40.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.669+0000 7f450affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:40.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.669+0000 7f450affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:40.670 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.669+0000 7f450affd640 10 monclient(hunting): dump: 2024-03-25T18:14:40.670 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:40.670 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:40.670 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:40.670 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:40.670 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:40.670 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:40.670 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:40.670 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:40.670 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:40.671 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:40.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2768822190 0 0) 0x7f4504003520 con 0x7f451c1430c0 2024-03-25T18:14:40.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 >> v1:172.21.15.86:6789/0 conn(0x7f451c0a98f0 legacy=0x7f451c1367f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 >> v1:172.21.15.64:6789/0 conn(0x7f451c13f710 legacy=0x7f451c0ac7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f450affd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:40.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f450affd640 1 monclient: found mon.c 2024-03-25T18:14:40.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f450affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:40.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f450affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:14:40.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451c0ac380 con 0x7f451c1430c0 2024-03-25T18:14:40.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f450affd640 10 monclient: _finish_auth 0 2024-03-25T18:14:40.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f450affd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:40.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f450affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:40.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f4520fc6640 5 monclient: authenticate success, global_id 4457 2024-03-25T18:14:40.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f4520fc6640 10 monclient: _renew_subs 2024-03-25T18:14:40.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f4520fc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:14:40.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f4520fc6640 1 -- 172.21.15.64:0/4061246515 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f451c1381c0 con 0x7f451c1430c0 2024-03-25T18:14:40.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f4520fc6640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:40.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f4520fc6640 10 monclient: _renew_subs 2024-03-25T18:14:40.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f4520fc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:14:40.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f4520fc6640 1 -- 172.21.15.64:0/4061246515 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f451c1e36c0 con 0x7f451c1430c0 2024-03-25T18:14:40.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.686+0000 7f4520fc6640 1 librados: init done 2024-03-25T18:14:40.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.687+0000 7f4520fc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:40.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.687+0000 7f4520fc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:40.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.687+0000 7f4520fc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:14:40.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:40.687+0000 7f4520fc6640 1 -- 172.21.15.64:0/4061246515 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f451c0b3ef0 con 0x7f451c1430c0 2024-03-25T18:14:41.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.010+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4514003c20 con 0x7f451c1430c0 2024-03-25T18:14:41.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.010+0000 7f450affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:41.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.010+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4514003f70 con 0x7f451c1430c0 2024-03-25T18:14:41.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.010+0000 7f450affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:41.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.010+0000 7f450affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:41.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.010+0000 7f450affd640 10 monclient: dump: 2024-03-25T18:14:41.011 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:41.011 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:41.011 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:41.011 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:41.011 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:41.011 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:41.011 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:41.011 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:41.012 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:41.012 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:41.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.010+0000 7f4518d99640 4 set_mon_vals no callback set 2024-03-25T18:14:41.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.011+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f4514004190 con 0x7f451c1430c0 2024-03-25T18:14:41.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.011+0000 7f451a59c640 10 monclient: get_auth_request con 0x7f44fc06df70 auth_method 0 2024-03-25T18:14:41.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.011+0000 7f450affd640 10 client.4457.objecter ms_handle_connect 0x7f44fc06df70 2024-03-25T18:14:41.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.018+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f4514054bc0 con 0x7f451c1430c0 2024-03-25T18:14:41.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.018+0000 7f450affd640 10 client.4457.objecter ms_dispatch 0x7f451c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:41.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.018+0000 7f450affd640 3 client.4457.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:41.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.018+0000 7f450affd640 3 client.4457.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:41.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.018+0000 7f450affd640 20 client.4457.objecter dump_active .. 0 homeless 2024-03-25T18:14:41.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.018+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f45140506d0 con 0x7f451c1430c0 2024-03-25T18:14:41.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.018+0000 7f450affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:41.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.018+0000 7f450affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:41.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.267+0000 7f4520fc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:41.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.267+0000 7f4520fc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:41.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.267+0000 7f4520fc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:14:41.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.267+0000 7f4520fc6640 1 -- 172.21.15.64:0/4061246515 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f451c0627b0 con 0x7f451c1430c0 2024-03-25T18:14:41.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.319+0000 7f450affd640 1 -- 172.21.15.64:0/4061246515 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f4514052600 con 0x7f451c1430c0 2024-03-25T18:14:41.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.319+0000 7f450affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:41.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.319+0000 7f450affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:41.320 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:14:41.320 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:41.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.321+0000 7f4520fc6640 10 librados: watch_flush enter 2024-03-25T18:14:41.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.321+0000 7f4520fc6640 10 librados: watch_flush exit 2024-03-25T18:14:41.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.321+0000 7f4520fc6640 20 client.4457.objecter shutdown clearing up homeless session... 2024-03-25T18:14:41.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.321+0000 7f4520fc6640 10 client.4457.objecter successfully canceled tick 2024-03-25T18:14:41.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.321+0000 7f4520fc6640 1 -- 172.21.15.64:0/4061246515 >> v1:172.21.15.64:6800/2915543638 conn(0x7f44fc06df70 legacy=0x7f44fc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.321+0000 7f4520fc6640 10 monclient: shutdown 2024-03-25T18:14:41.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.321+0000 7f4520fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:41.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.321+0000 7f4520fc6640 1 -- 172.21.15.64:0/4061246515 >> v1:172.21.15.64:6790/0 conn(0x7f451c1430c0 legacy=0x7f451c138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.321+0000 7f4520fc6640 1 -- 172.21.15.64:0/4061246515 shutdown_connections 2024-03-25T18:14:41.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.321+0000 7f4520fc6640 1 -- 172.21.15.64:0/4061246515 >> 172.21.15.64:0/4061246515 conn(0x7f451c0b7ea0 msgr2=0x7f451c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:41.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.322+0000 7f4520fc6640 1 -- 172.21.15.64:0/4061246515 shutdown_connections 2024-03-25T18:14:41.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.322+0000 7f4520fc6640 1 -- 172.21.15.64:0/4061246515 wait complete. 2024-03-25T18:14:41.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:41.322+0000 7f4520fc6640 1 librados: shutdown 2024-03-25T18:14:41.333 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:48.139 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:48.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.241+0000 7fac35940640 10 monclient: get_monmap_and_config 2024-03-25T18:14:48.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.241+0000 7fac35940640 10 monclient: build_initial_monmap 2024-03-25T18:14:48.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.241+0000 7fac35940640 10 monclient: monmap: 2024-03-25T18:14:48.243 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:14:48.243 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:48.243 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:14:48.242687+0000 2024-03-25T18:14:48.243 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:14:48.242687+0000 2024-03-25T18:14:48.243 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:48.243 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:48.243 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:14:48.243 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:14:48.243 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:14:48.243 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:48.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.242+0000 7fac35940640 1 Processor -- start 2024-03-25T18:14:48.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.242+0000 7fac35940640 1 -- start start 2024-03-25T18:14:48.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.242+0000 7fac35940640 10 monclient: init 2024-03-25T18:14:48.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.243+0000 7fac35940640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:48.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.243+0000 7fac35940640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:48.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.243+0000 7fac35940640 10 monclient(hunting): picked mon.noname-a con 0x7fac300b4d60 addr v1:172.21.15.64:6789/0 2024-03-25T18:14:48.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.243+0000 7fac35940640 10 monclient(hunting): picked mon.noname-c con 0x7fac300b1730 addr v1:172.21.15.86:6789/0 2024-03-25T18:14:48.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.243+0000 7fac35940640 10 monclient(hunting): picked mon.noname-b con 0x7fac3014be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:14:48.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.243+0000 7fac35940640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac300af940 con 0x7fac300b4d60 2024-03-25T18:14:48.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.243+0000 7fac35940640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac300b0a70 con 0x7fac300b1730 2024-03-25T18:14:48.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.243+0000 7fac35940640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac300b0d50 con 0x7fac3014be90 2024-03-25T18:14:48.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.243+0000 7fac35940640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:48.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.243+0000 7fac35940640 10 monclient(hunting): authenticate will time out at 7274.481666s 2024-03-25T18:14:48.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.255+0000 7fac2effd640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fac300b4d60 0x7fac3013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:36814/0 (socket says 172.21.15.64:36814) 2024-03-25T18:14:48.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.255+0000 7fac2effd640 1 -- 172.21.15.64:0/2746334680 learned_addr learned my addr 172.21.15.64:0/2746334680 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:14:48.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.281+0000 7fac2e7fc640 1 -- 172.21.15.64:0/2746334680 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528154873 0 0) 0x7fac300af940 con 0x7fac300b4d60 2024-03-25T18:14:48.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.281+0000 7fac2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:48.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.281+0000 7fac2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:48.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.281+0000 7fac2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:48.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.281+0000 7fac2e7fc640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:14:48.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.281+0000 7fac2e7fc640 1 -- 172.21.15.64:0/2746334680 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac1c003520 con 0x7fac300b4d60 2024-03-25T18:14:48.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.283+0000 7fac2e7fc640 1 -- 172.21.15.64:0/2746334680 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503315689 0 0) 0x7fac300b0a70 con 0x7fac300b1730 2024-03-25T18:14:48.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.283+0000 7fac2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:48.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.283+0000 7fac2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:48.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.283+0000 7fac2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:48.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.283+0000 7fac2e7fc640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:14:48.284 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.283+0000 7fac2e7fc640 1 -- 172.21.15.64:0/2746334680 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac300af940 con 0x7fac300b1730 2024-03-25T18:14:48.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.288+0000 7fac2e7fc640 1 -- 172.21.15.64:0/2746334680 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3712677882 0 0) 0x7fac1c003520 con 0x7fac300b4d60 2024-03-25T18:14:48.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.288+0000 7fac2e7fc640 1 -- 172.21.15.64:0/2746334680 >> v1:172.21.15.64:6790/0 conn(0x7fac3014be90 legacy=0x7fac3014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.288+0000 7fac2e7fc640 1 -- 172.21.15.64:0/2746334680 >> v1:172.21.15.86:6789/0 conn(0x7fac300b1730 legacy=0x7fac300b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.288+0000 7fac2e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:48.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.288+0000 7fac2e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:14:48.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.288+0000 7fac2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:48.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.288+0000 7fac2e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:14:48.289 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.288+0000 7fac2e7fc640 1 -- 172.21.15.64:0/2746334680 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac300b1030 con 0x7fac300b4d60 2024-03-25T18:14:48.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.288+0000 7fac2e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:48.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.288+0000 7fac2e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:48.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.288+0000 7fac2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:48.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.288+0000 7fac2e7fc640 1 -- 172.21.15.64:0/2746334680 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fac20001690 con 0x7fac300b4d60 2024-03-25T18:14:48.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.288+0000 7fac2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:48.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.288+0000 7fac2e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:48.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.288+0000 7fac2e7fc640 10 monclient: dump: 2024-03-25T18:14:48.290 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:48.290 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:48.290 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:48.290 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:48.290 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:48.290 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:48.290 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:48.291 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:48.291 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:48.291 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:48.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.288+0000 7fac35940640 5 monclient: authenticate success, global_id 4455 2024-03-25T18:14:48.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.288+0000 7fac35940640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:48.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.310+0000 7fac2e7fc640 1 -- 172.21.15.64:0/2746334680 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac20001870 con 0x7fac300b4d60 2024-03-25T18:14:48.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.310+0000 7fac2e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:48.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.310+0000 7fac2e7fc640 1 -- 172.21.15.64:0/2746334680 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fac20001c80 con 0x7fac300b4d60 2024-03-25T18:14:48.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.310+0000 7fac2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:48.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.310+0000 7fac2e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:48.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.310+0000 7fac2e7fc640 10 monclient: dump: 2024-03-25T18:14:48.311 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:48.311 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:48.311 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:48.311 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:48.311 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:48.311 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:48.311 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:48.311 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:48.311 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.310+0000 7fac35940640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.310+0000 7fac35940640 4 set_mon_vals no callback set 2024-03-25T18:14:48.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.311+0000 7fac35940640 10 monclient: shutdown 2024-03-25T18:14:48.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.311+0000 7fac35940640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:48.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.311+0000 7fac35940640 1 -- 172.21.15.64:0/2746334680 >> v1:172.21.15.64:6789/0 conn(0x7fac300b4d60 legacy=0x7fac3013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.311+0000 7fac35940640 1 -- 172.21.15.64:0/2746334680 shutdown_connections 2024-03-25T18:14:48.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.311+0000 7fac35940640 1 -- 172.21.15.64:0/2746334680 >> 172.21.15.64:0/2746334680 conn(0x7fac300617a0 msgr2=0x7fac30061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:48.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.311+0000 7fac35940640 1 -- 172.21.15.64:0/2746334680 shutdown_connections 2024-03-25T18:14:48.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.311+0000 7fac35940640 1 -- 172.21.15.64:0/2746334680 wait complete. 2024-03-25T18:14:48.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.312+0000 7fac35940640 10 monclient: build_initial_monmap 2024-03-25T18:14:48.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.312+0000 7fac35940640 10 monclient: monmap: 2024-03-25T18:14:48.314 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:14:48.314 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:48.315 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:14:48.315 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:14:48.315 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:48.315 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:14:48.315 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:14:48.315 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:14:48.315 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:14:48.315 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:48.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.312+0000 7fac35940640 1 librados: starting msgr at 2024-03-25T18:14:48.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.312+0000 7fac35940640 1 librados: starting objecter 2024-03-25T18:14:48.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.312+0000 7fac35940640 1 Processor -- start 2024-03-25T18:14:48.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.312+0000 7fac35940640 1 -- start start 2024-03-25T18:14:48.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.312+0000 7fac35940640 1 librados: setting wanted keys 2024-03-25T18:14:48.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.312+0000 7fac35940640 1 librados: calling monclient init 2024-03-25T18:14:48.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.312+0000 7fac35940640 10 monclient: init 2024-03-25T18:14:48.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.313+0000 7fac35940640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:48.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.313+0000 7fac35940640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:48.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.313+0000 7fac35940640 10 monclient(hunting): picked mon.noname-c con 0x7fac300b1730 addr v1:172.21.15.64:6790/0 2024-03-25T18:14:48.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.313+0000 7fac35940640 10 monclient(hunting): picked mon.noname-b con 0x7fac300b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:14:48.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.313+0000 7fac35940640 10 monclient(hunting): picked mon.noname-a con 0x7fac3014be90 addr v1:172.21.15.64:6789/0 2024-03-25T18:14:48.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.313+0000 7fac35940640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3013eeb0 con 0x7fac3014be90 2024-03-25T18:14:48.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.313+0000 7fac35940640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac30137860 con 0x7fac300b4d60 2024-03-25T18:14:48.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.313+0000 7fac35940640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3013e900 con 0x7fac300b1730 2024-03-25T18:14:48.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.313+0000 7fac35940640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:48.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.313+0000 7fac35940640 10 monclient(hunting): authenticate will time out at 7274.551972s 2024-03-25T18:14:48.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.324+0000 7fac2effd640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fac300b1730 0x7fac3012ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:45982/0 (socket says 172.21.15.64:45982) 2024-03-25T18:14:48.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.324+0000 7fac2effd640 1 -- 172.21.15.64:0/339987402 learned_addr learned my addr 172.21.15.64:0/339987402 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:14:48.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.326+0000 7fac2cff9640 10 client.?.objecter ms_handle_connect 0x7fac300b1730 2024-03-25T18:14:48.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.326+0000 7fac2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:48.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.326+0000 7fac2cff9640 10 client.?.objecter ms_handle_connect 0x7fac300b4d60 2024-03-25T18:14:48.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.326+0000 7fac2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:48.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.331+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014634041 0 0) 0x7fac30137860 con 0x7fac300b4d60 2024-03-25T18:14:48.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.331+0000 7fac2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:48.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.331+0000 7fac2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:48.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.331+0000 7fac2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:48.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.331+0000 7fac2cff9640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:14:48.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.332+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac10004eb0 con 0x7fac300b4d60 2024-03-25T18:14:48.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.334+0000 7fac2cff9640 10 client.?.objecter ms_handle_connect 0x7fac3014be90 2024-03-25T18:14:48.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.334+0000 7fac2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:48.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.341+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394614581 0 0) 0x7fac3013e900 con 0x7fac300b1730 2024-03-25T18:14:48.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.341+0000 7fac2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:48.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.341+0000 7fac2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:48.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.341+0000 7fac2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:48.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.341+0000 7fac2cff9640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:14:48.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.342+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac30137860 con 0x7fac300b1730 2024-03-25T18:14:48.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.344+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 386624896 0 0) 0x7fac3013eeb0 con 0x7fac3014be90 2024-03-25T18:14:48.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.344+0000 7fac2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:48.345 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.344+0000 7fac2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:48.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.344+0000 7fac2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:48.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.344+0000 7fac2cff9640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:14:48.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.344+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac3013e900 con 0x7fac3014be90 2024-03-25T18:14:48.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.357+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 378708234 0 0) 0x7fac10004eb0 con 0x7fac300b4d60 2024-03-25T18:14:48.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.357+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac3013eeb0 con 0x7fac300b4d60 2024-03-25T18:14:48.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.357+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fac18001930 con 0x7fac300b4d60 2024-03-25T18:14:48.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.357+0000 7fac2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:48.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.357+0000 7fac2cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:48.358 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.357+0000 7fac2cff9640 10 monclient(hunting): dump: 2024-03-25T18:14:48.358 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:48.358 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:48.359 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:48.359 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:48.359 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:48.359 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:48.359 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:48.359 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:48.359 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:48.359 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:48.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.357+0000 7fac2cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.361+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4143826697 0 0) 0x7fac30137860 con 0x7fac300b1730 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.361+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac10004eb0 con 0x7fac300b1730 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.361+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fac20001b80 con 0x7fac300b1730 2024-03-25T18:14:48.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.361+0000 7fac2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:48.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.361+0000 7fac2cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:48.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.361+0000 7fac2cff9640 10 monclient(hunting): dump: 2024-03-25T18:14:48.363 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:48.363 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:48.363 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:48.363 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:48.363 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:48.363 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:48.363 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:48.363 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:48.363 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:48.363 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:48.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.363+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3358002387 0 0) 0x7fac3013e900 con 0x7fac3014be90 2024-03-25T18:14:48.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.363+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac30137860 con 0x7fac3014be90 2024-03-25T18:14:48.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.363+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fac28001a60 con 0x7fac3014be90 2024-03-25T18:14:48.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.363+0000 7fac2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:48.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.363+0000 7fac2cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:48.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.363+0000 7fac2cff9640 10 monclient(hunting): dump: 2024-03-25T18:14:48.365 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:48.365 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:48.365 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:48.365 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:48.365 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:48.365 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:48.365 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:48.365 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:48.365 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:48.365 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:48.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1875006365 0 0) 0x7fac3013eeb0 con 0x7fac300b4d60 2024-03-25T18:14:48.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 >> v1:172.21.15.64:6790/0 conn(0x7fac300b1730 legacy=0x7fac3012ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 >> v1:172.21.15.64:6789/0 conn(0x7fac3014be90 legacy=0x7fac30138520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac2cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:14:48.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac2cff9640 1 monclient: found mon.b 2024-03-25T18:14:48.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac2cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac2cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3013ea90 con 0x7fac300b4d60 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac2cff9640 10 monclient: _finish_auth 0 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac2cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac35940640 5 monclient: authenticate success, global_id 4456 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac35940640 10 monclient: _renew_subs 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac35940640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac35940640 1 -- 172.21.15.64:0/339987402 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac30136320 con 0x7fac300b4d60 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac35940640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac35940640 10 monclient: _renew_subs 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac35940640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac35940640 1 -- 172.21.15.64:0/339987402 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac30136670 con 0x7fac300b4d60 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.368+0000 7fac35940640 1 librados: init done 2024-03-25T18:14:48.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.369+0000 7fac35940640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:48.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.369+0000 7fac35940640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:48.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.369+0000 7fac35940640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:14:48.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.369+0000 7fac35940640 1 -- 172.21.15.64:0/339987402 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac3013ec80 con 0x7fac300b4d60 2024-03-25T18:14:48.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.648+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac18003610 con 0x7fac300b4d60 2024-03-25T18:14:48.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.648+0000 7fac2cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:48.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.648+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fac18003780 con 0x7fac300b4d60 2024-03-25T18:14:48.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.648+0000 7fac2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:48.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.648+0000 7fac2cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:48.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.648+0000 7fac2cff9640 10 monclient: dump: 2024-03-25T18:14:48.650 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:48.650 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:48.650 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:48.650 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:48.650 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:48.650 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:48.651 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:48.651 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:48.651 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:48.651 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:48.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.648+0000 7fac2dffb640 4 set_mon_vals no callback set 2024-03-25T18:14:48.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.650+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fac180197d0 con 0x7fac300b4d60 2024-03-25T18:14:48.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.650+0000 7fac2effd640 10 monclient: get_auth_request con 0x7fac1006e0f0 auth_method 0 2024-03-25T18:14:48.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.650+0000 7fac2cff9640 10 client.4456.objecter ms_handle_connect 0x7fac1006e0f0 2024-03-25T18:14:48.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.655+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fac180533e0 con 0x7fac300b4d60 2024-03-25T18:14:48.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.655+0000 7fac2cff9640 10 client.4456.objecter ms_dispatch 0x7fac30000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:48.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.655+0000 7fac2cff9640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:48.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.655+0000 7fac2cff9640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:48.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.655+0000 7fac2cff9640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-25T18:14:48.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.655+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac1804eef0 con 0x7fac300b4d60 2024-03-25T18:14:48.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.655+0000 7fac2cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:48.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.655+0000 7fac2cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:48.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.908+0000 7fac35940640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:48.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.908+0000 7fac35940640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:48.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.908+0000 7fac35940640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:14:48.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.908+0000 7fac35940640 1 -- 172.21.15.64:0/339987402 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac3013f060 con 0x7fac300b4d60 2024-03-25T18:14:48.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.943+0000 7fac2cff9640 1 -- 172.21.15.64:0/339987402 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fac3013f060 con 0x7fac300b4d60 2024-03-25T18:14:48.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.943+0000 7fac2cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:48.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.943+0000 7fac2cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:48.944 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:14:48.944 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:48.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.945+0000 7fac35940640 10 librados: watch_flush enter 2024-03-25T18:14:48.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.946+0000 7fac35940640 10 librados: watch_flush exit 2024-03-25T18:14:48.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.946+0000 7fac35940640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-25T18:14:48.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.946+0000 7fac35940640 10 client.4456.objecter successfully canceled tick 2024-03-25T18:14:48.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.946+0000 7fac35940640 1 -- 172.21.15.64:0/339987402 >> v1:172.21.15.64:6800/2915543638 conn(0x7fac1006e0f0 legacy=0x7fac100705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.946+0000 7fac35940640 10 monclient: shutdown 2024-03-25T18:14:48.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.946+0000 7fac35940640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:48.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.946+0000 7fac35940640 1 -- 172.21.15.64:0/339987402 >> v1:172.21.15.86:6789/0 conn(0x7fac300b4d60 legacy=0x7fac30137e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.946+0000 7fac35940640 1 -- 172.21.15.64:0/339987402 shutdown_connections 2024-03-25T18:14:48.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.946+0000 7fac35940640 1 -- 172.21.15.64:0/339987402 >> 172.21.15.64:0/339987402 conn(0x7fac300617a0 msgr2=0x7fac3013c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:48.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.946+0000 7fac35940640 1 -- 172.21.15.64:0/339987402 shutdown_connections 2024-03-25T18:14:48.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.946+0000 7fac35940640 1 -- 172.21.15.64:0/339987402 wait complete. 2024-03-25T18:14:48.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:48.946+0000 7fac35940640 1 librados: shutdown 2024-03-25T18:14:48.959 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:55.766 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:55.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.881+0000 7fbaf5a2e640 10 monclient: get_monmap_and_config 2024-03-25T18:14:55.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.881+0000 7fbaf5a2e640 10 monclient: build_initial_monmap 2024-03-25T18:14:55.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.881+0000 7fbaf5a2e640 10 monclient: monmap: 2024-03-25T18:14:55.882 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:14:55.882 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:55.882 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:14:55.882362+0000 2024-03-25T18:14:55.882 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:14:55.882362+0000 2024-03-25T18:14:55.882 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:55.882 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:55.883 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:14:55.883 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:14:55.883 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:14:55.883 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:55.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.882+0000 7fbaf5a2e640 1 Processor -- start 2024-03-25T18:14:55.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.882+0000 7fbaf5a2e640 1 -- start start 2024-03-25T18:14:55.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.882+0000 7fbaf5a2e640 10 monclient: init 2024-03-25T18:14:55.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.882+0000 7fbaf5a2e640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:55.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.882+0000 7fbaf5a2e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:55.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.882+0000 7fbaf5a2e640 10 monclient(hunting): picked mon.noname-b con 0x7fbaf00a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:14:55.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.883+0000 7fbaf5a2e640 10 monclient(hunting): picked mon.noname-a con 0x7fbaf013f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:14:55.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.883+0000 7fbaf5a2e640 10 monclient(hunting): picked mon.noname-c con 0x7fbaf01430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:14:55.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.883+0000 7fbaf5a2e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf0142520 con 0x7fbaf013f710 2024-03-25T18:14:55.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.883+0000 7fbaf5a2e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf00b28a0 con 0x7fbaf01430c0 2024-03-25T18:14:55.884 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.883+0000 7fbaf5a2e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf00b2a30 con 0x7fbaf00a98f0 2024-03-25T18:14:55.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.883+0000 7fbaf5a2e640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:55.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.883+0000 7fbaf5a2e640 10 monclient(hunting): authenticate will time out at 7282.121527s 2024-03-25T18:14:55.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.888+0000 7fbaef7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fbaf01430c0 0x7fbaf014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:41314/0 (socket says 172.21.15.64:41314) 2024-03-25T18:14:55.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.888+0000 7fbaef7fe640 1 -- 172.21.15.64:0/2776854940 learned_addr learned my addr 172.21.15.64:0/2776854940 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:14:55.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.901+0000 7fbaee7fc640 1 -- 172.21.15.64:0/2776854940 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753663465 0 0) 0x7fbaf00b28a0 con 0x7fbaf01430c0 2024-03-25T18:14:55.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.901+0000 7fbaee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:55.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.901+0000 7fbaee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:55.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.901+0000 7fbaee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:55.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.901+0000 7fbaee7fc640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:14:55.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.901+0000 7fbaee7fc640 1 -- 172.21.15.64:0/2776854940 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbadc003520 con 0x7fbaf01430c0 2024-03-25T18:14:55.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.911+0000 7fbaee7fc640 1 -- 172.21.15.64:0/2776854940 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142045378 0 0) 0x7fbaf0142520 con 0x7fbaf013f710 2024-03-25T18:14:55.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.911+0000 7fbaee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:55.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.911+0000 7fbaee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:55.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.911+0000 7fbaee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:55.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.911+0000 7fbaee7fc640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:14:55.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.911+0000 7fbaee7fc640 1 -- 172.21.15.64:0/2776854940 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaf00b28a0 con 0x7fbaf013f710 2024-03-25T18:14:55.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.930+0000 7fbaee7fc640 1 -- 172.21.15.64:0/2776854940 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2147274437 0 0) 0x7fbaf00b28a0 con 0x7fbaf013f710 2024-03-25T18:14:55.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.930+0000 7fbaee7fc640 1 -- 172.21.15.64:0/2776854940 >> v1:172.21.15.64:6790/0 conn(0x7fbaf00a98f0 legacy=0x7fbaf00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:55.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.930+0000 7fbaee7fc640 1 -- 172.21.15.64:0/2776854940 >> v1:172.21.15.86:6789/0 conn(0x7fbaf01430c0 legacy=0x7fbaf014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:55.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.930+0000 7fbaee7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:55.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.930+0000 7fbaee7fc640 1 monclient: found mon.noname-a 2024-03-25T18:14:55.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.930+0000 7fbaee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:55.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.930+0000 7fbaee7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:14:55.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.930+0000 7fbaee7fc640 1 -- 172.21.15.64:0/2776854940 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf00b44c0 con 0x7fbaf013f710 2024-03-25T18:14:55.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.930+0000 7fbaee7fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:55.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.930+0000 7fbaee7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:55.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.930+0000 7fbaee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:55.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.930+0000 7fbaee7fc640 1 -- 172.21.15.64:0/2776854940 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbad80015f0 con 0x7fbaf013f710 2024-03-25T18:14:55.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.930+0000 7fbaee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:55.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.930+0000 7fbaee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:55.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.930+0000 7fbaee7fc640 10 monclient: dump: 2024-03-25T18:14:55.932 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:55.932 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:55.932 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:55.933 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:55.933 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:55.933 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:55.933 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:55.933 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:55.933 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:55.933 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:55.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.930+0000 7fbaf5a2e640 5 monclient: authenticate success, global_id 4461 2024-03-25T18:14:55.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.930+0000 7fbaf5a2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:55.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.938+0000 7fbaee7fc640 1 -- 172.21.15.64:0/2776854940 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbad8001750 con 0x7fbaf013f710 2024-03-25T18:14:55.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.938+0000 7fbaee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:55.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.938+0000 7fbaee7fc640 1 -- 172.21.15.64:0/2776854940 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbad8001ba0 con 0x7fbaf013f710 2024-03-25T18:14:55.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.938+0000 7fbaf5a2e640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:55.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.938+0000 7fbaf5a2e640 4 set_mon_vals no callback set 2024-03-25T18:14:55.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.938+0000 7fbaee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:55.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.938+0000 7fbaee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:55.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.938+0000 7fbaee7fc640 10 monclient: dump: 2024-03-25T18:14:55.939 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:55.939 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:55.939 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:55.939 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:55.939 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:55.939 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:55.939 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:55.939 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:55.940 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:55.940 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:55.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.938+0000 7fbaf5a2e640 10 monclient: shutdown 2024-03-25T18:14:55.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.938+0000 7fbaf5a2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:55.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.938+0000 7fbaf5a2e640 1 -- 172.21.15.64:0/2776854940 >> v1:172.21.15.64:6789/0 conn(0x7fbaf013f710 legacy=0x7fbaf01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:55.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.939+0000 7fbaf5a2e640 1 -- 172.21.15.64:0/2776854940 shutdown_connections 2024-03-25T18:14:55.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.939+0000 7fbaf5a2e640 1 -- 172.21.15.64:0/2776854940 >> 172.21.15.64:0/2776854940 conn(0x7fbaf00b7ea0 msgr2=0x7fbaf00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:55.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.939+0000 7fbaf5a2e640 1 -- 172.21.15.64:0/2776854940 shutdown_connections 2024-03-25T18:14:55.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.939+0000 7fbaf5a2e640 1 -- 172.21.15.64:0/2776854940 wait complete. 2024-03-25T18:14:55.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.939+0000 7fbaf5a2e640 10 monclient: build_initial_monmap 2024-03-25T18:14:55.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.939+0000 7fbaf5a2e640 10 monclient: monmap: 2024-03-25T18:14:55.942 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:14:55.942 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:55.942 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:14:55.942 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:14:55.942 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:55.942 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:14:55.942 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:14:55.942 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:14:55.942 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:14:55.942 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:55.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.940+0000 7fbaf5a2e640 1 librados: starting msgr at 2024-03-25T18:14:55.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.940+0000 7fbaf5a2e640 1 librados: starting objecter 2024-03-25T18:14:55.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.940+0000 7fbaf5a2e640 1 Processor -- start 2024-03-25T18:14:55.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.940+0000 7fbaf5a2e640 1 -- start start 2024-03-25T18:14:55.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.940+0000 7fbaf5a2e640 1 librados: setting wanted keys 2024-03-25T18:14:55.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.940+0000 7fbaf5a2e640 1 librados: calling monclient init 2024-03-25T18:14:55.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.940+0000 7fbaf5a2e640 10 monclient: init 2024-03-25T18:14:55.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.941+0000 7fbaf5a2e640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:55.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.941+0000 7fbaf5a2e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:55.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.941+0000 7fbaf5a2e640 10 monclient(hunting): picked mon.noname-b con 0x7fbaf00a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:14:55.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.941+0000 7fbaf5a2e640 10 monclient(hunting): picked mon.noname-a con 0x7fbaf013f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:14:55.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.941+0000 7fbaf5a2e640 10 monclient(hunting): picked mon.noname-c con 0x7fbaf01430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:14:55.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.941+0000 7fbaf5a2e640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf00b7d00 con 0x7fbaf013f710 2024-03-25T18:14:55.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.941+0000 7fbaf5a2e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf0136cb0 con 0x7fbaf00a98f0 2024-03-25T18:14:55.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.941+0000 7fbaf5a2e640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf0135c20 con 0x7fbaf01430c0 2024-03-25T18:14:55.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.941+0000 7fbaf5a2e640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:55.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.941+0000 7fbaf5a2e640 10 monclient(hunting): authenticate will time out at 7282.180314s 2024-03-25T18:14:55.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.950+0000 7fbae67fc640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fbaf013f710 0x7fbaf012fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:41530/0 (socket says 172.21.15.64:41530) 2024-03-25T18:14:55.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.950+0000 7fbae67fc640 1 -- 172.21.15.64:0/774712668 learned_addr learned my addr 172.21.15.64:0/774712668 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:14:55.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.951+0000 7fbaecff9640 10 client.?.objecter ms_handle_connect 0x7fbaf013f710 2024-03-25T18:14:55.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.951+0000 7fbaecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:55.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.962+0000 7fbaecff9640 10 client.?.objecter ms_handle_connect 0x7fbaf00a98f0 2024-03-25T18:14:55.963 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.962+0000 7fbaecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:55.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.965+0000 7fbaecff9640 10 client.?.objecter ms_handle_connect 0x7fbaf01430c0 2024-03-25T18:14:55.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.965+0000 7fbaecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:55.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.965+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999100010 0 0) 0x7fbaf00b44c0 con 0x7fbaf013f710 2024-03-25T18:14:55.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.965+0000 7fbaecff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:55.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.965+0000 7fbaecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:55.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.965+0000 7fbaecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:55.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.965+0000 7fbaecff9640 10 monclient(hunting): my global_id is 4464 2024-03-25T18:14:55.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.965+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad0004eb0 con 0x7fbaf013f710 2024-03-25T18:14:55.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.975+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3968586922 0 0) 0x7fbad0004eb0 con 0x7fbaf013f710 2024-03-25T18:14:55.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.975+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbaf00b44c0 con 0x7fbaf013f710 2024-03-25T18:14:55.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.975+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbad80019c0 con 0x7fbaf013f710 2024-03-25T18:14:55.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.975+0000 7fbaecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:55.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.975+0000 7fbaecff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:55.976 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.975+0000 7fbaecff9640 10 monclient(hunting): dump: 2024-03-25T18:14:55.976 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:55.976 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:55.976 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:55.976 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:55.976 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:55.976 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:55.976 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:55.977 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:55.977 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:55.977 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:55.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.975+0000 7fbaecff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:55.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.976+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347375895 0 0) 0x7fbaf0136cb0 con 0x7fbaf00a98f0 2024-03-25T18:14:55.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.976+0000 7fbaecff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:55.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.976+0000 7fbaecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:55.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.976+0000 7fbaecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:55.977 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.976+0000 7fbaecff9640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:14:55.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.976+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad0004eb0 con 0x7fbaf00a98f0 2024-03-25T18:14:55.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.988+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3735681379 0 0) 0x7fbad0004eb0 con 0x7fbaf00a98f0 2024-03-25T18:14:55.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.988+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbaf0136cb0 con 0x7fbaf00a98f0 2024-03-25T18:14:55.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.988+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbae0001830 con 0x7fbaf00a98f0 2024-03-25T18:14:55.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.988+0000 7fbaecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:55.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.988+0000 7fbaecff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:55.989 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.988+0000 7fbaecff9640 10 monclient(hunting): dump: 2024-03-25T18:14:55.990 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:55.990 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:55.990 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:55.990 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:55.990 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:55.990 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:55.990 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:55.990 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:55.990 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:55.990 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:55.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.992+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932108797 0 0) 0x7fbaf0135c20 con 0x7fbaf01430c0 2024-03-25T18:14:55.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.992+0000 7fbaecff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:55.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.992+0000 7fbaecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:55.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.992+0000 7fbaecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:55.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.992+0000 7fbaecff9640 10 monclient(hunting): my global_id is 4469 2024-03-25T18:14:55.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:55.992+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad0004eb0 con 0x7fbaf01430c0 2024-03-25T18:14:56.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1986636322 0 0) 0x7fbaf00b44c0 con 0x7fbaf013f710 2024-03-25T18:14:56.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 >> v1:172.21.15.64:6790/0 conn(0x7fbaf01430c0 legacy=0x7fbaf01304b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:56.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 >> v1:172.21.15.86:6789/0 conn(0x7fbaf00a98f0 legacy=0x7fbaf0135510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:56.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaecff9640 10 monclient: _finish_hunting 0 2024-03-25T18:14:56.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaecff9640 1 monclient: found mon.a 2024-03-25T18:14:56.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:56.002 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaecff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:14:56.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf0130bc0 con 0x7fbaf013f710 2024-03-25T18:14:56.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaecff9640 10 monclient: _finish_auth 0 2024-03-25T18:14:56.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaecff9640 10 monclient: _check_auth_tickets 2024-03-25T18:14:56.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:56.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaf5a2e640 5 monclient: authenticate success, global_id 4464 2024-03-25T18:14:56.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaf5a2e640 10 monclient: _renew_subs 2024-03-25T18:14:56.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaf5a2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:14:56.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaf5a2e640 1 -- 172.21.15.64:0/774712668 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf0136f00 con 0x7fbaf013f710 2024-03-25T18:14:56.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaf5a2e640 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:56.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaf5a2e640 10 monclient: _renew_subs 2024-03-25T18:14:56.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaf5a2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:14:56.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaf5a2e640 1 -- 172.21.15.64:0/774712668 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf01374c0 con 0x7fbaf013f710 2024-03-25T18:14:56.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.001+0000 7fbaf5a2e640 1 librados: init done 2024-03-25T18:14:56.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.002+0000 7fbaf5a2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:56.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.002+0000 7fbaf5a2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:56.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.002+0000 7fbaf5a2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:14:56.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.002+0000 7fbaf5a2e640 1 -- 172.21.15.64:0/774712668 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf00627b0 con 0x7fbaf013f710 2024-03-25T18:14:56.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.327+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbad80038a0 con 0x7fbaf013f710 2024-03-25T18:14:56.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.327+0000 7fbaecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:56.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.327+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbad8001250 con 0x7fbaf013f710 2024-03-25T18:14:56.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.327+0000 7fbaecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:56.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.327+0000 7fbaecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:56.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.327+0000 7fbaecff9640 10 monclient: dump: 2024-03-25T18:14:56.329 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:14:56.329 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:14:56.330 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:56.330 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:14:56.330 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:56.330 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:14:56.330 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:14:56.330 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:14:56.330 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:14:56.330 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:14:56.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.327+0000 7fbaedffb640 4 set_mon_vals no callback set 2024-03-25T18:14:56.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.328+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fbad8003d30 con 0x7fbaf013f710 2024-03-25T18:14:56.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.329+0000 7fbaeeffd640 10 monclient: get_auth_request con 0x7fbad006dfb0 auth_method 0 2024-03-25T18:14:56.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.329+0000 7fbaecff9640 10 client.4464.objecter ms_handle_connect 0x7fbad006dfb0 2024-03-25T18:14:56.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.334+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fbad80547a0 con 0x7fbaf013f710 2024-03-25T18:14:56.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.334+0000 7fbaecff9640 10 client.4464.objecter ms_dispatch 0x7fbaf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:56.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.334+0000 7fbaecff9640 3 client.4464.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:56.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.334+0000 7fbaecff9640 3 client.4464.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:56.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.334+0000 7fbaecff9640 20 client.4464.objecter dump_active .. 0 homeless 2024-03-25T18:14:56.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.334+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbad80502b0 con 0x7fbaf013f710 2024-03-25T18:14:56.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.334+0000 7fbaecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:56.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.334+0000 7fbaecff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:56.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.582+0000 7fbaf5a2e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:56.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.582+0000 7fbaf5a2e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:56.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.582+0000 7fbaf5a2e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:14:56.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.582+0000 7fbaf5a2e640 1 -- 172.21.15.64:0/774712668 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbaf0074cc0 con 0x7fbaf013f710 2024-03-25T18:14:56.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.612+0000 7fbaecff9640 1 -- 172.21.15.64:0/774712668 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fbad80521e0 con 0x7fbaf013f710 2024-03-25T18:14:56.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.612+0000 7fbaecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:56.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.612+0000 7fbaecff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:56.613 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:14:56.614 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:56.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.615+0000 7fbaf5a2e640 10 librados: watch_flush enter 2024-03-25T18:14:56.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.615+0000 7fbaf5a2e640 10 librados: watch_flush exit 2024-03-25T18:14:56.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.615+0000 7fbaf5a2e640 20 client.4464.objecter shutdown clearing up homeless session... 2024-03-25T18:14:56.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.615+0000 7fbaf5a2e640 10 client.4464.objecter successfully canceled tick 2024-03-25T18:14:56.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.615+0000 7fbaf5a2e640 1 -- 172.21.15.64:0/774712668 >> v1:172.21.15.64:6800/2915543638 conn(0x7fbad006dfb0 legacy=0x7fbad0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:56.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.615+0000 7fbaf5a2e640 10 monclient: shutdown 2024-03-25T18:14:56.616 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.615+0000 7fbaf5a2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:56.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.615+0000 7fbaf5a2e640 1 -- 172.21.15.64:0/774712668 >> v1:172.21.15.64:6789/0 conn(0x7fbaf013f710 legacy=0x7fbaf012fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:56.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.615+0000 7fbaf5a2e640 1 -- 172.21.15.64:0/774712668 shutdown_connections 2024-03-25T18:14:56.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.615+0000 7fbaf5a2e640 1 -- 172.21.15.64:0/774712668 >> 172.21.15.64:0/774712668 conn(0x7fbaf00b7ea0 msgr2=0x7fbaf0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:56.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.615+0000 7fbaf5a2e640 1 -- 172.21.15.64:0/774712668 shutdown_connections 2024-03-25T18:14:56.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.615+0000 7fbaf5a2e640 1 -- 172.21.15.64:0/774712668 wait complete. 2024-03-25T18:14:56.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:14:56.615+0000 7fbaf5a2e640 1 librados: shutdown 2024-03-25T18:14:56.627 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:57.438 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:05:40.550 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:57.445 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:05:40.558 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:57.499 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:05:40.611 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:57.499 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:05:40.611 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:57.506 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:05:40.619 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:57.512 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:05:40.591 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:57.520 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:05:40.598 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:57.534 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:05:40.646 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:57.541 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:05:40.619 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:03.433 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:03.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.546+0000 7fdcc5c63640 10 monclient: get_monmap_and_config 2024-03-25T18:15:03.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.546+0000 7fdcc5c63640 10 monclient: build_initial_monmap 2024-03-25T18:15:03.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.546+0000 7fdcc5c63640 10 monclient: monmap: 2024-03-25T18:15:03.548 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:15:03.548 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:03.548 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:15:03.547724+0000 2024-03-25T18:15:03.548 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:15:03.547724+0000 2024-03-25T18:15:03.548 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:03.548 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:03.548 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:15:03.548 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:15:03.548 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:15:03.548 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:03.549 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.547+0000 7fdcc5c63640 1 Processor -- start 2024-03-25T18:15:03.549 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.547+0000 7fdcc5c63640 1 -- start start 2024-03-25T18:15:03.549 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.547+0000 7fdcc5c63640 10 monclient: init 2024-03-25T18:15:03.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.548+0000 7fdcc5c63640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:03.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.548+0000 7fdcc5c63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:03.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.548+0000 7fdcc5c63640 10 monclient(hunting): picked mon.noname-b con 0x7fdcc00a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:15:03.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.548+0000 7fdcc5c63640 10 monclient(hunting): picked mon.noname-a con 0x7fdcc013f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:15:03.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.548+0000 7fdcc5c63640 10 monclient(hunting): picked mon.noname-c con 0x7fdcc01430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:15:03.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.548+0000 7fdcc5c63640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc0142520 con 0x7fdcc013f710 2024-03-25T18:15:03.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.548+0000 7fdcc5c63640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc00b28a0 con 0x7fdcc01430c0 2024-03-25T18:15:03.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.548+0000 7fdcc5c63640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc00b2a30 con 0x7fdcc00a98f0 2024-03-25T18:15:03.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.548+0000 7fdcc5c63640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:03.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.548+0000 7fdcc5c63640 10 monclient(hunting): authenticate will time out at 7289.787143s 2024-03-25T18:15:03.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.558+0000 7fdcbffff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fdcc01430c0 0x7fdcc014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:59746/0 (socket says 172.21.15.64:59746) 2024-03-25T18:15:03.559 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.558+0000 7fdcbffff640 1 -- 172.21.15.64:0/3149998362 learned_addr learned my addr 172.21.15.64:0/3149998362 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:15:03.569 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.569+0000 7fdcbe7fc640 1 -- 172.21.15.64:0/3149998362 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458642154 0 0) 0x7fdcc0142520 con 0x7fdcc013f710 2024-03-25T18:15:03.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.569+0000 7fdcbe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:03.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.569+0000 7fdcbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:03.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.569+0000 7fdcbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:03.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.569+0000 7fdcbe7fc640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:15:03.570 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.569+0000 7fdcbe7fc640 1 -- 172.21.15.64:0/3149998362 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca8003520 con 0x7fdcc013f710 2024-03-25T18:15:03.577 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 1 -- 172.21.15.64:0/3149998362 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421210701 0 0) 0x7fdcc00b2a30 con 0x7fdcc00a98f0 2024-03-25T18:15:03.577 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:03.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:03.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:03.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:15:03.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 1 -- 172.21.15.64:0/3149998362 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc0142520 con 0x7fdcc00a98f0 2024-03-25T18:15:03.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 1 -- 172.21.15.64:0/3149998362 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1409135891 0 0) 0x7fdca8003520 con 0x7fdcc013f710 2024-03-25T18:15:03.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 1 -- 172.21.15.64:0/3149998362 >> v1:172.21.15.64:6790/0 conn(0x7fdcc00a98f0 legacy=0x7fdcc00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:03.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 1 -- 172.21.15.64:0/3149998362 >> v1:172.21.15.86:6789/0 conn(0x7fdcc01430c0 legacy=0x7fdcc014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:03.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:03.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 1 monclient: found mon.noname-a 2024-03-25T18:15:03.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:03.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:15:03.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 1 -- 172.21.15.64:0/3149998362 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc00b44c0 con 0x7fdcc013f710 2024-03-25T18:15:03.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:03.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:03.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:03.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 1 -- 172.21.15.64:0/3149998362 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fdcb4001690 con 0x7fdcc013f710 2024-03-25T18:15:03.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:03.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:03.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcbe7fc640 10 monclient: dump: 2024-03-25T18:15:03.579 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:03.579 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:03.579 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:03.579 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:03.579 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:03.579 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:03.579 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:03.579 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:03.579 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:03.579 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:03.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcc5c63640 5 monclient: authenticate success, global_id 4467 2024-03-25T18:15:03.580 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.576+0000 7fdcc5c63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:03.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.582+0000 7fdcbe7fc640 1 -- 172.21.15.64:0/3149998362 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcb40017f0 con 0x7fdcc013f710 2024-03-25T18:15:03.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.582+0000 7fdcbe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:03.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.582+0000 7fdcbe7fc640 1 -- 172.21.15.64:0/3149998362 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fdcb4001c00 con 0x7fdcc013f710 2024-03-25T18:15:03.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.582+0000 7fdcbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:03.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.582+0000 7fdcbe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:03.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.582+0000 7fdcbe7fc640 10 monclient: dump: 2024-03-25T18:15:03.583 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:03.583 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:03.583 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:03.583 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:03.584 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:03.584 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:03.584 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:03.584 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:03.584 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:03.584 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:03.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.582+0000 7fdcc5c63640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:03.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.582+0000 7fdcc5c63640 4 set_mon_vals no callback set 2024-03-25T18:15:03.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.583+0000 7fdcc5c63640 10 monclient: shutdown 2024-03-25T18:15:03.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.583+0000 7fdcc5c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:03.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.583+0000 7fdcc5c63640 1 -- 172.21.15.64:0/3149998362 >> v1:172.21.15.64:6789/0 conn(0x7fdcc013f710 legacy=0x7fdcc01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:03.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.583+0000 7fdcc5c63640 1 -- 172.21.15.64:0/3149998362 shutdown_connections 2024-03-25T18:15:03.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.583+0000 7fdcc5c63640 1 -- 172.21.15.64:0/3149998362 >> 172.21.15.64:0/3149998362 conn(0x7fdcc00b7ea0 msgr2=0x7fdcc00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:03.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.583+0000 7fdcc5c63640 1 -- 172.21.15.64:0/3149998362 shutdown_connections 2024-03-25T18:15:03.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.583+0000 7fdcc5c63640 1 -- 172.21.15.64:0/3149998362 wait complete. 2024-03-25T18:15:03.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.584+0000 7fdcc5c63640 10 monclient: build_initial_monmap 2024-03-25T18:15:03.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.584+0000 7fdcc5c63640 10 monclient: monmap: 2024-03-25T18:15:03.586 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:15:03.586 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:03.586 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:15:03.586 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:15:03.586 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:03.586 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:15:03.586 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:15:03.586 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:15:03.587 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:15:03.587 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:03.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.584+0000 7fdcc5c63640 1 librados: starting msgr at 2024-03-25T18:15:03.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.584+0000 7fdcc5c63640 1 librados: starting objecter 2024-03-25T18:15:03.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.584+0000 7fdcc5c63640 1 Processor -- start 2024-03-25T18:15:03.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.585+0000 7fdcc5c63640 1 -- start start 2024-03-25T18:15:03.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.585+0000 7fdcc5c63640 1 librados: setting wanted keys 2024-03-25T18:15:03.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.585+0000 7fdcc5c63640 1 librados: calling monclient init 2024-03-25T18:15:03.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.585+0000 7fdcc5c63640 10 monclient: init 2024-03-25T18:15:03.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.585+0000 7fdcc5c63640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:03.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.585+0000 7fdcc5c63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:03.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.585+0000 7fdcc5c63640 10 monclient(hunting): picked mon.noname-c con 0x7fdcc00a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:15:03.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.585+0000 7fdcc5c63640 10 monclient(hunting): picked mon.noname-a con 0x7fdcc013f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:15:03.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.585+0000 7fdcc5c63640 10 monclient(hunting): picked mon.noname-b con 0x7fdcc01430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:15:03.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.585+0000 7fdcc5c63640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc01397c0 con 0x7fdcc013f710 2024-03-25T18:15:03.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.585+0000 7fdcc5c63640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc01daa20 con 0x7fdcc01430c0 2024-03-25T18:15:03.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.585+0000 7fdcc5c63640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc01d8210 con 0x7fdcc00a98f0 2024-03-25T18:15:03.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.585+0000 7fdcc5c63640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:03.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.585+0000 7fdcc5c63640 10 monclient(hunting): authenticate will time out at 7289.824217s 2024-03-25T18:15:03.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.590+0000 7fdcbeffd640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fdcc013f710 0x7fdcc01d73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35954/0 (socket says 172.21.15.64:35954) 2024-03-25T18:15:03.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.590+0000 7fdcbeffd640 1 -- 172.21.15.64:0/403714768 learned_addr learned my addr 172.21.15.64:0/403714768 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:15:03.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.594+0000 7fdca7fff640 10 client.?.objecter ms_handle_connect 0x7fdcc013f710 2024-03-25T18:15:03.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.594+0000 7fdca7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:03.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.601+0000 7fdca7fff640 10 client.?.objecter ms_handle_connect 0x7fdcc01430c0 2024-03-25T18:15:03.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.601+0000 7fdca7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:03.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.606+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724502257 0 0) 0x7fdcc01397c0 con 0x7fdcc013f710 2024-03-25T18:15:03.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.606+0000 7fdca7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:03.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.606+0000 7fdca7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:03.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.606+0000 7fdca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:03.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.606+0000 7fdca7fff640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:15:03.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.606+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcac004eb0 con 0x7fdcc013f710 2024-03-25T18:15:03.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.607+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928817703 0 0) 0x7fdcc01daa20 con 0x7fdcc01430c0 2024-03-25T18:15:03.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.607+0000 7fdca7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:03.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.607+0000 7fdca7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:03.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.607+0000 7fdca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:03.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.607+0000 7fdca7fff640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:15:03.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.607+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc01397c0 con 0x7fdcc01430c0 2024-03-25T18:15:03.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.613+0000 7fdca7fff640 10 client.?.objecter ms_handle_connect 0x7fdcc00a98f0 2024-03-25T18:15:03.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.613+0000 7fdca7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:03.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.621+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4269397175 0 0) 0x7fdcac004eb0 con 0x7fdcc013f710 2024-03-25T18:15:03.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.621+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcc01daa20 con 0x7fdcc013f710 2024-03-25T18:15:03.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.621+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fdcb4001690 con 0x7fdcc013f710 2024-03-25T18:15:03.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.621+0000 7fdca7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:03.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.621+0000 7fdca7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:03.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.621+0000 7fdca7fff640 10 monclient(hunting): dump: 2024-03-25T18:15:03.622 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:03.622 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:03.623 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:03.623 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:03.623 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:03.623 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:03.623 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:03.623 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:03.623 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:03.623 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:03.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.621+0000 7fdca7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:03.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.625+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890118596 0 0) 0x7fdcc01d8210 con 0x7fdcc00a98f0 2024-03-25T18:15:03.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.625+0000 7fdca7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:03.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.625+0000 7fdca7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:03.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.625+0000 7fdca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:03.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.625+0000 7fdca7fff640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:15:03.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.625+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcac004eb0 con 0x7fdcc00a98f0 2024-03-25T18:15:03.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3804955001 0 0) 0x7fdcc01daa20 con 0x7fdcc013f710 2024-03-25T18:15:03.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 >> v1:172.21.15.64:6790/0 conn(0x7fdcc00a98f0 legacy=0x7fdcc01daf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:03.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 >> v1:172.21.15.86:6789/0 conn(0x7fdcc01430c0 legacy=0x7fdcc01d7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:03.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdca7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:03.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdca7fff640 1 monclient: found mon.a 2024-03-25T18:15:03.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdca7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:03.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdca7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc01db650 con 0x7fdcc013f710 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdca7fff640 10 monclient: _finish_auth 0 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdca7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdca7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdcc5c63640 5 monclient: authenticate success, global_id 4470 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdcc5c63640 10 monclient: _renew_subs 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdcc5c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdcc5c63640 1 -- 172.21.15.64:0/403714768 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc0137720 con 0x7fdcc013f710 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdcc5c63640 10 client.4470.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdcc5c63640 10 monclient: _renew_subs 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdcc5c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdcc5c63640 1 -- 172.21.15.64:0/403714768 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc0137d50 con 0x7fdcc013f710 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.632+0000 7fdcc5c63640 1 librados: init done 2024-03-25T18:15:03.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.633+0000 7fdcc5c63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:03.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.633+0000 7fdcc5c63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:03.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.633+0000 7fdcc5c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:15:03.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.633+0000 7fdcc5c63640 1 -- 172.21.15.64:0/403714768 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc01db840 con 0x7fdcc013f710 2024-03-25T18:15:03.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.937+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcb40038a0 con 0x7fdcc013f710 2024-03-25T18:15:03.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.937+0000 7fdca7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:03.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.937+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fdcb4003ad0 con 0x7fdcc013f710 2024-03-25T18:15:03.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.937+0000 7fdca7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:03.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.937+0000 7fdca7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:03.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.937+0000 7fdca7fff640 10 monclient: dump: 2024-03-25T18:15:03.939 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:03.939 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:03.939 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:03.939 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:03.939 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:03.939 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:03.939 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:03.939 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:03.939 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:03.939 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:03.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.937+0000 7fdcbdffb640 4 set_mon_vals no callback set 2024-03-25T18:15:03.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.938+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fdcb4003cf0 con 0x7fdcc013f710 2024-03-25T18:15:03.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.938+0000 7fdcbf7fe640 10 monclient: get_auth_request con 0x7fdcac06de00 auth_method 0 2024-03-25T18:15:03.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.939+0000 7fdca7fff640 10 client.4470.objecter ms_handle_connect 0x7fdcac06de00 2024-03-25T18:15:03.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.946+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fdcb40547e0 con 0x7fdcc013f710 2024-03-25T18:15:03.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.946+0000 7fdca7fff640 10 client.4470.objecter ms_dispatch 0x7fdcc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:03.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.946+0000 7fdca7fff640 3 client.4470.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:03.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.946+0000 7fdca7fff640 3 client.4470.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:03.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.946+0000 7fdca7fff640 20 client.4470.objecter dump_active .. 0 homeless 2024-03-25T18:15:03.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.946+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdcb40502f0 con 0x7fdcc013f710 2024-03-25T18:15:03.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.946+0000 7fdca7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:03.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:03.946+0000 7fdca7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:04.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.175+0000 7fdcc5c63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:04.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.175+0000 7fdcc5c63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:04.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.175+0000 7fdcc5c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:15:04.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.175+0000 7fdcc5c63640 1 -- 172.21.15.64:0/403714768 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcc01dabb0 con 0x7fdcc013f710 2024-03-25T18:15:04.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.236+0000 7fdca7fff640 1 -- 172.21.15.64:0/403714768 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fdcb4052220 con 0x7fdcc013f710 2024-03-25T18:15:04.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.236+0000 7fdca7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:04.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.236+0000 7fdca7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:04.237 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:15:04.237 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:04.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.238+0000 7fdcc5c63640 10 librados: watch_flush enter 2024-03-25T18:15:04.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.238+0000 7fdcc5c63640 10 librados: watch_flush exit 2024-03-25T18:15:04.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.238+0000 7fdcc5c63640 20 client.4470.objecter shutdown clearing up homeless session... 2024-03-25T18:15:04.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.238+0000 7fdcc5c63640 10 client.4470.objecter successfully canceled tick 2024-03-25T18:15:04.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.239+0000 7fdcc5c63640 1 -- 172.21.15.64:0/403714768 >> v1:172.21.15.64:6800/2915543638 conn(0x7fdcac06de00 legacy=0x7fdcac0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:04.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.239+0000 7fdcc5c63640 10 monclient: shutdown 2024-03-25T18:15:04.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.239+0000 7fdcc5c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:04.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.239+0000 7fdcc5c63640 1 -- 172.21.15.64:0/403714768 >> v1:172.21.15.64:6789/0 conn(0x7fdcc013f710 legacy=0x7fdcc01d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:04.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.239+0000 7fdcc5c63640 1 -- 172.21.15.64:0/403714768 shutdown_connections 2024-03-25T18:15:04.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.239+0000 7fdcc5c63640 1 -- 172.21.15.64:0/403714768 >> 172.21.15.64:0/403714768 conn(0x7fdcc00b7ea0 msgr2=0x7fdcc013c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:04.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.239+0000 7fdcc5c63640 1 -- 172.21.15.64:0/403714768 shutdown_connections 2024-03-25T18:15:04.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.239+0000 7fdcc5c63640 1 -- 172.21.15.64:0/403714768 wait complete. 2024-03-25T18:15:04.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:04.239+0000 7fdcc5c63640 1 librados: shutdown 2024-03-25T18:15:04.250 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:11.056 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:11.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.171+0000 7f4acc92b640 10 monclient: get_monmap_and_config 2024-03-25T18:15:11.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.171+0000 7f4acc92b640 10 monclient: build_initial_monmap 2024-03-25T18:15:11.174 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.171+0000 7f4acc92b640 10 monclient: monmap: 2024-03-25T18:15:11.174 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:15:11.174 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:11.174 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:15:11.173824+0000 2024-03-25T18:15:11.174 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:15:11.173824+0000 2024-03-25T18:15:11.175 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:11.175 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:11.175 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:15:11.175 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:15:11.175 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:15:11.175 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:11.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.172+0000 7f4acc92b640 1 Processor -- start 2024-03-25T18:15:11.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.172+0000 7f4acc92b640 1 -- start start 2024-03-25T18:15:11.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.172+0000 7f4acc92b640 10 monclient: init 2024-03-25T18:15:11.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.173+0000 7f4acc92b640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:11.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.173+0000 7f4acc92b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:11.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.173+0000 7f4acc92b640 10 monclient(hunting): picked mon.noname-a con 0x7f4ac80a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:15:11.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.173+0000 7f4acc92b640 10 monclient(hunting): picked mon.noname-b con 0x7f4ac813f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:15:11.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.173+0000 7f4acc92b640 10 monclient(hunting): picked mon.noname-c con 0x7f4ac81430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:15:11.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.173+0000 7f4acc92b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac8142520 con 0x7f4ac80a98f0 2024-03-25T18:15:11.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.173+0000 7f4acc92b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac80b28a0 con 0x7f4ac81430c0 2024-03-25T18:15:11.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.173+0000 7f4acc92b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac80b2a30 con 0x7f4ac813f710 2024-03-25T18:15:11.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.173+0000 7f4acc92b640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:11.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.173+0000 7f4acc92b640 10 monclient(hunting): authenticate will time out at 7297.413417s 2024-03-25T18:15:11.184 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.182+0000 7f4ac5d9b640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f4ac813f710 0x7f4ac81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53518/0 (socket says 172.21.15.64:53518) 2024-03-25T18:15:11.184 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.182+0000 7f4ac5d9b640 1 -- 172.21.15.64:0/565751717 learned_addr learned my addr 172.21.15.64:0/565751717 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:15:11.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.198+0000 7f4ac559a640 1 -- 172.21.15.64:0/565751717 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683255135 0 0) 0x7f4ac80b2a30 con 0x7f4ac813f710 2024-03-25T18:15:11.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.198+0000 7f4ac559a640 10 monclient(hunting): _negotiate 2024-03-25T18:15:11.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.198+0000 7f4ac559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:11.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.198+0000 7f4ac559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:11.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.198+0000 7f4ac559a640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:15:11.200 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.198+0000 7f4ac559a640 1 -- 172.21.15.64:0/565751717 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ab0003520 con 0x7f4ac813f710 2024-03-25T18:15:11.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 1 -- 172.21.15.64:0/565751717 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247585575 0 0) 0x7f4ac8142520 con 0x7f4ac80a98f0 2024-03-25T18:15:11.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 10 monclient(hunting): _negotiate 2024-03-25T18:15:11.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:11.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:11.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:15:11.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 1 -- 172.21.15.64:0/565751717 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac80b2a30 con 0x7f4ac80a98f0 2024-03-25T18:15:11.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 1 -- 172.21.15.64:0/565751717 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2658805032 0 0) 0x7f4ab0003520 con 0x7f4ac813f710 2024-03-25T18:15:11.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 1 -- 172.21.15.64:0/565751717 >> v1:172.21.15.86:6789/0 conn(0x7f4ac81430c0 legacy=0x7f4ac814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:11.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 1 -- 172.21.15.64:0/565751717 >> v1:172.21.15.64:6789/0 conn(0x7f4ac80a98f0 legacy=0x7f4ac80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:11.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 10 monclient: _finish_hunting 0 2024-03-25T18:15:11.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 1 monclient: found mon.noname-b 2024-03-25T18:15:11.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:11.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:15:11.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 1 -- 172.21.15.64:0/565751717 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac80b44c0 con 0x7f4ac813f710 2024-03-25T18:15:11.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 10 monclient: _finish_auth 0 2024-03-25T18:15:11.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 10 monclient: _check_auth_tickets 2024-03-25T18:15:11.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:11.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 1 -- 172.21.15.64:0/565751717 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4abc001690 con 0x7f4ac813f710 2024-03-25T18:15:11.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:11.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:11.211 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4ac559a640 10 monclient: dump: 2024-03-25T18:15:11.211 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:11.211 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:11.211 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:11.211 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:11.211 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:11.211 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:11.211 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:11.212 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:11.212 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:11.212 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:11.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4acc92b640 5 monclient: authenticate success, global_id 4478 2024-03-25T18:15:11.212 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.207+0000 7f4acc92b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:11.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.221+0000 7f4ac559a640 1 -- 172.21.15.64:0/565751717 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4abc0017f0 con 0x7f4ac813f710 2024-03-25T18:15:11.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.221+0000 7f4ac559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:11.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.221+0000 7f4ac559a640 1 -- 172.21.15.64:0/565751717 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4abc001c00 con 0x7f4ac813f710 2024-03-25T18:15:11.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.221+0000 7f4acc92b640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:11.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.221+0000 7f4acc92b640 4 set_mon_vals no callback set 2024-03-25T18:15:11.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.221+0000 7f4ac559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:11.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.221+0000 7f4ac559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:11.224 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.221+0000 7f4ac559a640 10 monclient: dump: 2024-03-25T18:15:11.224 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:11.224 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:11.224 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:11.224 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:11.224 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:11.224 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:11.224 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:11.224 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:11.224 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:11.224 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:11.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.222+0000 7f4acc92b640 10 monclient: shutdown 2024-03-25T18:15:11.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.222+0000 7f4acc92b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:11.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.222+0000 7f4acc92b640 1 -- 172.21.15.64:0/565751717 >> v1:172.21.15.64:6790/0 conn(0x7f4ac813f710 legacy=0x7f4ac81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:11.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.222+0000 7f4acc92b640 1 -- 172.21.15.64:0/565751717 shutdown_connections 2024-03-25T18:15:11.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.222+0000 7f4acc92b640 1 -- 172.21.15.64:0/565751717 >> 172.21.15.64:0/565751717 conn(0x7f4ac80b7ea0 msgr2=0x7f4ac80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:11.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.222+0000 7f4acc92b640 1 -- 172.21.15.64:0/565751717 shutdown_connections 2024-03-25T18:15:11.226 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.222+0000 7f4acc92b640 1 -- 172.21.15.64:0/565751717 wait complete. 2024-03-25T18:15:11.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.223+0000 7f4acc92b640 10 monclient: build_initial_monmap 2024-03-25T18:15:11.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.223+0000 7f4acc92b640 10 monclient: monmap: 2024-03-25T18:15:11.227 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:15:11.227 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:11.227 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:15:11.227 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:15:11.227 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:11.227 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:15:11.227 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:15:11.227 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:15:11.227 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:15:11.227 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:11.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.224+0000 7f4acc92b640 1 librados: starting msgr at 2024-03-25T18:15:11.227 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.224+0000 7f4acc92b640 1 librados: starting objecter 2024-03-25T18:15:11.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.224+0000 7f4acc92b640 1 Processor -- start 2024-03-25T18:15:11.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.224+0000 7f4acc92b640 1 -- start start 2024-03-25T18:15:11.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.224+0000 7f4acc92b640 1 librados: setting wanted keys 2024-03-25T18:15:11.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.224+0000 7f4acc92b640 1 librados: calling monclient init 2024-03-25T18:15:11.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.224+0000 7f4acc92b640 10 monclient: init 2024-03-25T18:15:11.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.225+0000 7f4acc92b640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:11.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.225+0000 7f4acc92b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:11.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.225+0000 7f4acc92b640 10 monclient(hunting): picked mon.noname-a con 0x7f4ac80a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:15:11.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.225+0000 7f4acc92b640 10 monclient(hunting): picked mon.noname-b con 0x7f4ac813f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:15:11.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.225+0000 7f4acc92b640 10 monclient(hunting): picked mon.noname-c con 0x7f4ac81430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:15:11.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.225+0000 7f4acc92b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac8139a30 con 0x7f4ac80a98f0 2024-03-25T18:15:11.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.225+0000 7f4acc92b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac81dabc0 con 0x7f4ac813f710 2024-03-25T18:15:11.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.225+0000 7f4acc92b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac81d41e0 con 0x7f4ac81430c0 2024-03-25T18:15:11.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.225+0000 7f4acc92b640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:11.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.225+0000 7f4acc92b640 10 monclient(hunting): authenticate will time out at 7297.464815s 2024-03-25T18:15:11.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.234+0000 7f4ac659c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f4ac80a98f0 0x7f4ac8138930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:48940/0 (socket says 172.21.15.64:48940) 2024-03-25T18:15:11.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.234+0000 7f4ac659c640 1 -- 172.21.15.64:0/2709580845 learned_addr learned my addr 172.21.15.64:0/2709580845 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:15:11.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.235+0000 7f4ab6ffd640 10 client.?.objecter ms_handle_connect 0x7f4ac80a98f0 2024-03-25T18:15:11.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.235+0000 7f4ab6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:11.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.242+0000 7f4ab6ffd640 10 client.?.objecter ms_handle_connect 0x7f4ac81430c0 2024-03-25T18:15:11.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.242+0000 7f4ab6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:11.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.244+0000 7f4ab6ffd640 10 client.?.objecter ms_handle_connect 0x7f4ac813f710 2024-03-25T18:15:11.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.244+0000 7f4ab6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:11.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.254+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814269135 0 0) 0x7f4ac8139a30 con 0x7f4ac80a98f0 2024-03-25T18:15:11.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.254+0000 7f4ab6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:11.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.254+0000 7f4ab6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:11.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.254+0000 7f4ab6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:11.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.254+0000 7f4ab6ffd640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:15:11.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.254+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa8004eb0 con 0x7f4ac80a98f0 2024-03-25T18:15:11.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.264+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1201418048 0 0) 0x7f4aa8004eb0 con 0x7f4ac80a98f0 2024-03-25T18:15:11.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.264+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ac8139a30 con 0x7f4ac80a98f0 2024-03-25T18:15:11.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.264+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4ab8001a10 con 0x7f4ac80a98f0 2024-03-25T18:15:11.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.264+0000 7f4ab6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:11.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.264+0000 7f4ab6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:11.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.264+0000 7f4ab6ffd640 10 monclient(hunting): dump: 2024-03-25T18:15:11.266 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:11.266 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:11.266 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:11.266 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:11.266 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:11.267 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:11.267 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:11.267 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:11.267 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:11.267 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:11.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.264+0000 7f4ab6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:11.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.265+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3519550541 0 0) 0x7f4ac81dabc0 con 0x7f4ac813f710 2024-03-25T18:15:11.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.265+0000 7f4ab6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:11.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.265+0000 7f4ab6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:11.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.265+0000 7f4ab6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:11.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.265+0000 7f4ab6ffd640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:15:11.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.265+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa8004eb0 con 0x7f4ac813f710 2024-03-25T18:15:11.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.270+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532500330 0 0) 0x7f4ac81d41e0 con 0x7f4ac81430c0 2024-03-25T18:15:11.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.270+0000 7f4ab6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:11.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.270+0000 7f4ab6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:11.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.270+0000 7f4ab6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:11.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.270+0000 7f4ab6ffd640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:15:11.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.270+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac81dabc0 con 0x7f4ac81430c0 2024-03-25T18:15:11.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4134039992 0 0) 0x7f4ac8139a30 con 0x7f4ac80a98f0 2024-03-25T18:15:11.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 >> v1:172.21.15.64:6790/0 conn(0x7f4ac81430c0 legacy=0x7f4ac81d4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:11.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 >> v1:172.21.15.86:6789/0 conn(0x7f4ac813f710 legacy=0x7f4ac81db0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:11.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4ab6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:11.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4ab6ffd640 1 monclient: found mon.a 2024-03-25T18:15:11.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4ab6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:11.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4ab6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:15:11.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac81d4370 con 0x7f4ac80a98f0 2024-03-25T18:15:11.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4ab6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:15:11.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4ab6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:11.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4ab6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:11.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4acc92b640 5 monclient: authenticate success, global_id 4476 2024-03-25T18:15:11.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4acc92b640 10 monclient: _renew_subs 2024-03-25T18:15:11.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4acc92b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:15:11.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4acc92b640 1 -- 172.21.15.64:0/2709580845 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac8139040 con 0x7f4ac80a98f0 2024-03-25T18:15:11.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4acc92b640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:11.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4acc92b640 10 monclient: _renew_subs 2024-03-25T18:15:11.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4acc92b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:15:11.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4acc92b640 1 -- 172.21.15.64:0/2709580845 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac81db990 con 0x7f4ac80a98f0 2024-03-25T18:15:11.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.278+0000 7f4acc92b640 1 librados: init done 2024-03-25T18:15:11.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.279+0000 7f4acc92b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:11.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.279+0000 7f4acc92b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:11.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.279+0000 7f4acc92b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:15:11.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.279+0000 7f4acc92b640 1 -- 172.21.15.64:0/2709580845 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac80627b0 con 0x7f4ac80a98f0 2024-03-25T18:15:11.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.587+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ab8001c90 con 0x7f4ac80a98f0 2024-03-25T18:15:11.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.587+0000 7f4ab6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:11.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.587+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4ab80038c0 con 0x7f4ac80a98f0 2024-03-25T18:15:11.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.587+0000 7f4ab6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:11.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.587+0000 7f4ab6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:11.590 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.587+0000 7f4ab6ffd640 10 monclient: dump: 2024-03-25T18:15:11.590 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:11.590 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:11.590 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:11.590 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:11.590 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:11.590 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:11.591 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:11.591 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:11.591 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:11.591 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:11.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.587+0000 7f4ac4d99640 4 set_mon_vals no callback set 2024-03-25T18:15:11.591 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.588+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f4ab8003ae0 con 0x7f4ac80a98f0 2024-03-25T18:15:11.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.588+0000 7f4ac5d9b640 10 monclient: get_auth_request con 0x7f4aa806de80 auth_method 0 2024-03-25T18:15:11.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.589+0000 7f4ab6ffd640 10 client.4476.objecter ms_handle_connect 0x7f4aa806de80 2024-03-25T18:15:11.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.594+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f4ab8054700 con 0x7f4ac80a98f0 2024-03-25T18:15:11.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.594+0000 7f4ab6ffd640 10 client.4476.objecter ms_dispatch 0x7f4ac8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:11.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.594+0000 7f4ab6ffd640 3 client.4476.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:11.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.594+0000 7f4ab6ffd640 3 client.4476.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:11.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.595+0000 7f4ab6ffd640 20 client.4476.objecter dump_active .. 0 homeless 2024-03-25T18:15:11.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.595+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ab8050210 con 0x7f4ac80a98f0 2024-03-25T18:15:11.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.595+0000 7f4ab6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:11.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.595+0000 7f4ab6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:11.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.857+0000 7f4acc92b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:11.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.858+0000 7f4acc92b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:11.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.858+0000 7f4acc92b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:15:11.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.858+0000 7f4acc92b640 1 -- 172.21.15.64:0/2709580845 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ac813dfd0 con 0x7f4ac80a98f0 2024-03-25T18:15:11.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.892+0000 7f4ab6ffd640 1 -- 172.21.15.64:0/2709580845 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f4ab8052140 con 0x7f4ac80a98f0 2024-03-25T18:15:11.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.892+0000 7f4ab6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:11.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.892+0000 7f4ab6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:11.893 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:15:11.893 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:11.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.894+0000 7f4acc92b640 10 librados: watch_flush enter 2024-03-25T18:15:11.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.894+0000 7f4acc92b640 10 librados: watch_flush exit 2024-03-25T18:15:11.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.894+0000 7f4acc92b640 20 client.4476.objecter shutdown clearing up homeless session... 2024-03-25T18:15:11.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.894+0000 7f4acc92b640 10 client.4476.objecter successfully canceled tick 2024-03-25T18:15:11.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.894+0000 7f4acc92b640 1 -- 172.21.15.64:0/2709580845 >> v1:172.21.15.64:6800/2915543638 conn(0x7f4aa806de80 legacy=0x7f4aa8070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:11.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.894+0000 7f4acc92b640 10 monclient: shutdown 2024-03-25T18:15:11.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.894+0000 7f4acc92b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:11.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.894+0000 7f4acc92b640 1 -- 172.21.15.64:0/2709580845 >> v1:172.21.15.64:6789/0 conn(0x7f4ac80a98f0 legacy=0x7f4ac8138930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:11.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.894+0000 7f4acc92b640 1 -- 172.21.15.64:0/2709580845 shutdown_connections 2024-03-25T18:15:11.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.894+0000 7f4acc92b640 1 -- 172.21.15.64:0/2709580845 >> 172.21.15.64:0/2709580845 conn(0x7f4ac80b7ea0 msgr2=0x7f4ac8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:11.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.894+0000 7f4acc92b640 1 -- 172.21.15.64:0/2709580845 shutdown_connections 2024-03-25T18:15:11.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.894+0000 7f4acc92b640 1 -- 172.21.15.64:0/2709580845 wait complete. 2024-03-25T18:15:11.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:11.894+0000 7f4acc92b640 1 librados: shutdown 2024-03-25T18:15:11.907 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:17.420 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:06:00.532 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:17.469 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:06:00.581 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:17.472 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:06:00.584 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:17.495 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:06:00.607 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:17.510 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:06:00.623 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:17.518 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:06:00.597 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:17.527 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:06:00.640 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:17.534 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:06:00.613 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:17.558 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:06:00.637 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:18.714 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:18.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.829+0000 7ff881cfb640 10 monclient: get_monmap_and_config 2024-03-25T18:15:18.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.829+0000 7ff881cfb640 10 monclient: build_initial_monmap 2024-03-25T18:15:18.830 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.829+0000 7ff881cfb640 10 monclient: monmap: 2024-03-25T18:15:18.830 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:15:18.830 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:18.830 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:15:18.830213+0000 2024-03-25T18:15:18.830 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:15:18.830213+0000 2024-03-25T18:15:18.830 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:18.830 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:18.830 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:15:18.831 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:15:18.831 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:15:18.831 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:18.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.830+0000 7ff881cfb640 1 Processor -- start 2024-03-25T18:15:18.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.830+0000 7ff881cfb640 1 -- start start 2024-03-25T18:15:18.831 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.830+0000 7ff881cfb640 10 monclient: init 2024-03-25T18:15:18.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.831+0000 7ff881cfb640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:18.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.831+0000 7ff881cfb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:18.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.831+0000 7ff881cfb640 10 monclient(hunting): picked mon.noname-b con 0x7ff87c0b4d60 addr v1:172.21.15.64:6790/0 2024-03-25T18:15:18.832 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.831+0000 7ff881cfb640 10 monclient(hunting): picked mon.noname-c con 0x7ff87c0b1730 addr v1:172.21.15.86:6789/0 2024-03-25T18:15:18.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.831+0000 7ff881cfb640 10 monclient(hunting): picked mon.noname-a con 0x7ff87c14be90 addr v1:172.21.15.64:6789/0 2024-03-25T18:15:18.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.831+0000 7ff881cfb640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff87c0af940 con 0x7ff87c14be90 2024-03-25T18:15:18.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.831+0000 7ff881cfb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff87c0b0a70 con 0x7ff87c0b1730 2024-03-25T18:15:18.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.831+0000 7ff881cfb640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff87c0b0d50 con 0x7ff87c0b4d60 2024-03-25T18:15:18.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.831+0000 7ff881cfb640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:18.833 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.831+0000 7ff881cfb640 10 monclient(hunting): authenticate will time out at 7305.069917s 2024-03-25T18:15:18.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.841+0000 7ff87bfff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7ff87c14be90 0x7ff87c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:33078/0 (socket says 172.21.15.64:33078) 2024-03-25T18:15:18.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.841+0000 7ff87bfff640 1 -- 172.21.15.64:0/3288744542 learned_addr learned my addr 172.21.15.64:0/3288744542 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:15:18.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.853+0000 7ff87a7fc640 1 -- 172.21.15.64:0/3288744542 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218380875 0 0) 0x7ff87c0b0a70 con 0x7ff87c0b1730 2024-03-25T18:15:18.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.853+0000 7ff87a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:18.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.853+0000 7ff87a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:18.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.853+0000 7ff87a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:18.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.853+0000 7ff87a7fc640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:15:18.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.853+0000 7ff87a7fc640 1 -- 172.21.15.64:0/3288744542 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff864003520 con 0x7ff87c0b1730 2024-03-25T18:15:18.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.857+0000 7ff87a7fc640 1 -- 172.21.15.64:0/3288744542 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804097843 0 0) 0x7ff87c0af940 con 0x7ff87c14be90 2024-03-25T18:15:18.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.857+0000 7ff87a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:18.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.857+0000 7ff87a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:18.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.857+0000 7ff87a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:18.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.857+0000 7ff87a7fc640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:15:18.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.857+0000 7ff87a7fc640 1 -- 172.21.15.64:0/3288744542 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff87c0b0a70 con 0x7ff87c14be90 2024-03-25T18:15:18.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.861+0000 7ff87a7fc640 1 -- 172.21.15.64:0/3288744542 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 606849007 0 0) 0x7ff864003520 con 0x7ff87c0b1730 2024-03-25T18:15:18.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.861+0000 7ff87a7fc640 1 -- 172.21.15.64:0/3288744542 >> v1:172.21.15.64:6790/0 conn(0x7ff87c0b4d60 legacy=0x7ff87c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:18.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.861+0000 7ff87a7fc640 1 -- 172.21.15.64:0/3288744542 >> v1:172.21.15.64:6789/0 conn(0x7ff87c14be90 legacy=0x7ff87c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:18.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.861+0000 7ff87a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:18.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.861+0000 7ff87a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:15:18.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.861+0000 7ff87a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:18.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.861+0000 7ff87a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:15:18.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.861+0000 7ff87a7fc640 1 -- 172.21.15.64:0/3288744542 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c0b1030 con 0x7ff87c0b1730 2024-03-25T18:15:18.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.861+0000 7ff87a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:18.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.861+0000 7ff87a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:18.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.861+0000 7ff87a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:18.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.861+0000 7ff87a7fc640 1 -- 172.21.15.64:0/3288744542 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ff870001690 con 0x7ff87c0b1730 2024-03-25T18:15:18.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.861+0000 7ff87a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:18.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.861+0000 7ff87a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:18.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.861+0000 7ff87a7fc640 10 monclient: dump: 2024-03-25T18:15:18.864 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:18.864 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:18.864 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:18.864 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:18.864 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:18.865 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:18.865 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:18.865 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:18.865 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:18.865 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:18.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.861+0000 7ff881cfb640 5 monclient: authenticate success, global_id 4477 2024-03-25T18:15:18.865 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.861+0000 7ff881cfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:18.874 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.874+0000 7ff87a7fc640 1 -- 172.21.15.64:0/3288744542 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8700017f0 con 0x7ff87c0b1730 2024-03-25T18:15:18.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.874+0000 7ff87a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:18.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.874+0000 7ff87a7fc640 1 -- 172.21.15.64:0/3288744542 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ff870001c00 con 0x7ff87c0b1730 2024-03-25T18:15:18.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.874+0000 7ff87a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:18.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.874+0000 7ff87a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:18.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.874+0000 7ff87a7fc640 10 monclient: dump: 2024-03-25T18:15:18.875 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:18.875 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:18.875 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:18.875 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:18.875 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:18.875 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:18.875 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:18.875 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:18.875 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:18.875 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:18.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.874+0000 7ff881cfb640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:18.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.874+0000 7ff881cfb640 4 set_mon_vals no callback set 2024-03-25T18:15:18.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.875+0000 7ff881cfb640 10 monclient: shutdown 2024-03-25T18:15:18.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.875+0000 7ff881cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:18.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.875+0000 7ff881cfb640 1 -- 172.21.15.64:0/3288744542 >> v1:172.21.15.86:6789/0 conn(0x7ff87c0b1730 legacy=0x7ff87c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:18.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.875+0000 7ff881cfb640 1 -- 172.21.15.64:0/3288744542 shutdown_connections 2024-03-25T18:15:18.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.875+0000 7ff881cfb640 1 -- 172.21.15.64:0/3288744542 >> 172.21.15.64:0/3288744542 conn(0x7ff87c0617a0 msgr2=0x7ff87c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:18.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.875+0000 7ff881cfb640 1 -- 172.21.15.64:0/3288744542 shutdown_connections 2024-03-25T18:15:18.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.875+0000 7ff881cfb640 1 -- 172.21.15.64:0/3288744542 wait complete. 2024-03-25T18:15:18.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.875+0000 7ff881cfb640 10 monclient: build_initial_monmap 2024-03-25T18:15:18.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.875+0000 7ff881cfb640 10 monclient: monmap: 2024-03-25T18:15:18.879 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:15:18.879 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:18.879 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:15:18.879 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:15:18.879 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:18.879 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:15:18.880 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:15:18.880 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:15:18.880 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:15:18.880 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:18.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.876+0000 7ff881cfb640 1 librados: starting msgr at 2024-03-25T18:15:18.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.876+0000 7ff881cfb640 1 librados: starting objecter 2024-03-25T18:15:18.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.876+0000 7ff881cfb640 1 Processor -- start 2024-03-25T18:15:18.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.876+0000 7ff881cfb640 1 -- start start 2024-03-25T18:15:18.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.876+0000 7ff881cfb640 1 librados: setting wanted keys 2024-03-25T18:15:18.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.876+0000 7ff881cfb640 1 librados: calling monclient init 2024-03-25T18:15:18.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.876+0000 7ff881cfb640 10 monclient: init 2024-03-25T18:15:18.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.876+0000 7ff881cfb640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:18.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.876+0000 7ff881cfb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:18.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.876+0000 7ff881cfb640 10 monclient(hunting): picked mon.noname-a con 0x7ff87c0b1730 addr v1:172.21.15.64:6789/0 2024-03-25T18:15:18.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.876+0000 7ff881cfb640 10 monclient(hunting): picked mon.noname-b con 0x7ff87c0b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:15:18.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.876+0000 7ff881cfb640 10 monclient(hunting): picked mon.noname-c con 0x7ff87c14be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:15:18.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.876+0000 7ff881cfb640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff87c13bfb0 con 0x7ff87c0b1730 2024-03-25T18:15:18.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.876+0000 7ff881cfb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff87c1458e0 con 0x7ff87c0b4d60 2024-03-25T18:15:18.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.876+0000 7ff881cfb640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff87c12f7b0 con 0x7ff87c14be90 2024-03-25T18:15:18.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.876+0000 7ff881cfb640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:18.882 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.876+0000 7ff881cfb640 10 monclient(hunting): authenticate will time out at 7305.115429s 2024-03-25T18:15:18.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.891+0000 7ff87affd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff87c0b4d60 0x7ff87c12e990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:52188/0 (socket says 172.21.15.64:52188) 2024-03-25T18:15:18.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.891+0000 7ff87affd640 1 -- 172.21.15.64:0/2117053238 learned_addr learned my addr 172.21.15.64:0/2117053238 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:15:18.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.891+0000 7ff863fff640 10 client.?.objecter ms_handle_connect 0x7ff87c0b4d60 2024-03-25T18:15:18.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.891+0000 7ff863fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:18.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.899+0000 7ff863fff640 10 client.?.objecter ms_handle_connect 0x7ff87c14be90 2024-03-25T18:15:18.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.899+0000 7ff863fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:18.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.901+0000 7ff863fff640 10 client.?.objecter ms_handle_connect 0x7ff87c0b1730 2024-03-25T18:15:18.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.901+0000 7ff863fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:18.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.905+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265928899 0 0) 0x7ff87c1458e0 con 0x7ff87c0b4d60 2024-03-25T18:15:18.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.905+0000 7ff863fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:18.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.905+0000 7ff863fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:18.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.905+0000 7ff863fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:18.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.905+0000 7ff863fff640 10 monclient(hunting): my global_id is 4480 2024-03-25T18:15:18.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.905+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff868004eb0 con 0x7ff87c0b4d60 2024-03-25T18:15:18.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.909+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835635827 0 0) 0x7ff87c13bfb0 con 0x7ff87c0b1730 2024-03-25T18:15:18.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.909+0000 7ff863fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:18.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.909+0000 7ff863fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:18.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.909+0000 7ff863fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:18.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.909+0000 7ff863fff640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:15:18.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.909+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff87c1458e0 con 0x7ff87c0b1730 2024-03-25T18:15:18.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.912+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182195338 0 0) 0x7ff87c12f7b0 con 0x7ff87c14be90 2024-03-25T18:15:18.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.912+0000 7ff863fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:18.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.912+0000 7ff863fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:18.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.912+0000 7ff863fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:18.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.912+0000 7ff863fff640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:15:18.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.912+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff87c13bfb0 con 0x7ff87c14be90 2024-03-25T18:15:18.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.926+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3622840605 0 0) 0x7ff868004eb0 con 0x7ff87c0b4d60 2024-03-25T18:15:18.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.926+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff87c12f7b0 con 0x7ff87c0b4d60 2024-03-25T18:15:18.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.926+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ff870001690 con 0x7ff87c0b4d60 2024-03-25T18:15:18.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.926+0000 7ff863fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:18.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.926+0000 7ff863fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:18.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.926+0000 7ff863fff640 10 monclient(hunting): dump: 2024-03-25T18:15:18.927 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:18.927 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:18.928 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:18.928 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:18.928 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:18.928 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:18.928 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:18.928 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:18.928 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:18.928 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:18.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.926+0000 7ff863fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:18.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.926+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3433675329 0 0) 0x7ff87c1458e0 con 0x7ff87c0b1730 2024-03-25T18:15:18.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.926+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff868004eb0 con 0x7ff87c0b1730 2024-03-25T18:15:18.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.926+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ff8740018a0 con 0x7ff87c0b1730 2024-03-25T18:15:18.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.926+0000 7ff863fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:18.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.926+0000 7ff863fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:18.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.926+0000 7ff863fff640 10 monclient(hunting): dump: 2024-03-25T18:15:18.928 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:18.928 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:18.929 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:18.929 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:18.929 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:18.929 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:18.929 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:18.929 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:18.929 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:18.929 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:18.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 471254059 0 0) 0x7ff87c12f7b0 con 0x7ff87c0b4d60 2024-03-25T18:15:18.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 >> v1:172.21.15.64:6790/0 conn(0x7ff87c14be90 legacy=0x7ff87c12f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:18.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 >> v1:172.21.15.64:6789/0 conn(0x7ff87c0b1730 legacy=0x7ff87c13b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:18.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff863fff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:18.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff863fff640 1 monclient: found mon.b 2024-03-25T18:15:18.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff863fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:18.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff863fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:18.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c13aa90 con 0x7ff87c0b4d60 2024-03-25T18:15:18.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff863fff640 10 monclient: _finish_auth 0 2024-03-25T18:15:18.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff863fff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:18.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff863fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:18.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff881cfb640 5 monclient: authenticate success, global_id 4480 2024-03-25T18:15:18.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff881cfb640 10 monclient: _renew_subs 2024-03-25T18:15:18.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff881cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:18.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff881cfb640 1 -- 172.21.15.64:0/2117053238 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff87c13ad70 con 0x7ff87c0b4d60 2024-03-25T18:15:18.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff881cfb640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:18.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff881cfb640 10 monclient: _renew_subs 2024-03-25T18:15:18.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff881cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:18.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff881cfb640 1 -- 172.21.15.64:0/2117053238 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff87c145330 con 0x7ff87c0b4d60 2024-03-25T18:15:18.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.940+0000 7ff881cfb640 1 librados: init done 2024-03-25T18:15:18.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.942+0000 7ff881cfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:18.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.942+0000 7ff881cfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:18.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.942+0000 7ff881cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:18.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:18.942+0000 7ff881cfb640 1 -- 172.21.15.64:0/2117053238 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff87c1401f0 con 0x7ff87c0b4d60 2024-03-25T18:15:19.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.158+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8700038a0 con 0x7ff87c0b4d60 2024-03-25T18:15:19.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.158+0000 7ff863fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:19.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.158+0000 7ff879ffb640 4 set_mon_vals no callback set 2024-03-25T18:15:19.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.215+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ff870001cc0 con 0x7ff87c0b4d60 2024-03-25T18:15:19.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.215+0000 7ff863fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:19.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.215+0000 7ff863fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:19.216 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.215+0000 7ff863fff640 10 monclient: dump: 2024-03-25T18:15:19.216 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:19.217 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:19.217 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:19.217 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:19.217 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:19.217 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:19.217 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:19.217 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:19.217 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:19.217 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:19.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.216+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7ff870003d70 con 0x7ff87c0b4d60 2024-03-25T18:15:19.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.217+0000 7ff87b7fe640 10 monclient: get_auth_request con 0x7ff86806df70 auth_method 0 2024-03-25T18:15:19.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.217+0000 7ff863fff640 10 client.4480.objecter ms_handle_connect 0x7ff86806df70 2024-03-25T18:15:19.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.222+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7ff8700549e0 con 0x7ff87c0b4d60 2024-03-25T18:15:19.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.222+0000 7ff863fff640 10 client.4480.objecter ms_dispatch 0x7ff87c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:19.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.222+0000 7ff863fff640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:19.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.222+0000 7ff863fff640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:19.223 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.222+0000 7ff863fff640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-25T18:15:19.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.227+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff870050320 con 0x7ff87c0b4d60 2024-03-25T18:15:19.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.227+0000 7ff863fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:19.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.227+0000 7ff863fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:19.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.478+0000 7ff881cfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:19.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.478+0000 7ff881cfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:19.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.478+0000 7ff881cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:19.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.478+0000 7ff881cfb640 1 -- 172.21.15.64:0/2117053238 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff87c0afbd0 con 0x7ff87c0b4d60 2024-03-25T18:15:19.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.502+0000 7ff863fff640 1 -- 172.21.15.64:0/2117053238 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7ff870052420 con 0x7ff87c0b4d60 2024-03-25T18:15:19.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.502+0000 7ff863fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:19.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.502+0000 7ff863fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:19.503 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:15:19.503 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:19.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.505+0000 7ff881cfb640 10 librados: watch_flush enter 2024-03-25T18:15:19.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.505+0000 7ff881cfb640 10 librados: watch_flush exit 2024-03-25T18:15:19.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.505+0000 7ff881cfb640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-25T18:15:19.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.505+0000 7ff881cfb640 10 client.4480.objecter successfully canceled tick 2024-03-25T18:15:19.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.505+0000 7ff881cfb640 1 -- 172.21.15.64:0/2117053238 >> v1:172.21.15.64:6800/2915543638 conn(0x7ff86806df70 legacy=0x7ff868070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:19.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.505+0000 7ff881cfb640 10 monclient: shutdown 2024-03-25T18:15:19.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.505+0000 7ff881cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:19.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.505+0000 7ff881cfb640 1 -- 172.21.15.64:0/2117053238 >> v1:172.21.15.86:6789/0 conn(0x7ff87c0b4d60 legacy=0x7ff87c12e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:19.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.505+0000 7ff881cfb640 1 -- 172.21.15.64:0/2117053238 shutdown_connections 2024-03-25T18:15:19.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.505+0000 7ff881cfb640 1 -- 172.21.15.64:0/2117053238 >> 172.21.15.64:0/2117053238 conn(0x7ff87c0617a0 msgr2=0x7ff87c13c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:19.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.505+0000 7ff881cfb640 1 -- 172.21.15.64:0/2117053238 shutdown_connections 2024-03-25T18:15:19.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.505+0000 7ff881cfb640 1 -- 172.21.15.64:0/2117053238 wait complete. 2024-03-25T18:15:19.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:19.505+0000 7ff881cfb640 1 librados: shutdown 2024-03-25T18:15:19.518 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:26.324 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:26.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.436+0000 7fd1ff130640 10 monclient: get_monmap_and_config 2024-03-25T18:15:26.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.436+0000 7fd1ff130640 10 monclient: build_initial_monmap 2024-03-25T18:15:26.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.436+0000 7fd1ff130640 10 monclient: monmap: 2024-03-25T18:15:26.438 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:15:26.438 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:26.438 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:15:26.437841+0000 2024-03-25T18:15:26.438 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:15:26.437841+0000 2024-03-25T18:15:26.438 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:26.438 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:26.438 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:15:26.438 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:15:26.438 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:15:26.439 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:26.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.438+0000 7fd1ff130640 1 Processor -- start 2024-03-25T18:15:26.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.438+0000 7fd1ff130640 1 -- start start 2024-03-25T18:15:26.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.438+0000 7fd1ff130640 10 monclient: init 2024-03-25T18:15:26.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.439+0000 7fd1ff130640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:26.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.439+0000 7fd1ff130640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:26.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.439+0000 7fd1ff130640 10 monclient(hunting): picked mon.noname-c con 0x7fd1f80a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:15:26.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.439+0000 7fd1ff130640 10 monclient(hunting): picked mon.noname-b con 0x7fd1f813f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:15:26.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.439+0000 7fd1ff130640 10 monclient(hunting): picked mon.noname-a con 0x7fd1f81430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:15:26.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.439+0000 7fd1ff130640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1f8142520 con 0x7fd1f81430c0 2024-03-25T18:15:26.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.439+0000 7fd1ff130640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1f80b28a0 con 0x7fd1f80a98f0 2024-03-25T18:15:26.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.439+0000 7fd1ff130640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1f80b2a30 con 0x7fd1f813f710 2024-03-25T18:15:26.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.439+0000 7fd1ff130640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:26.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.439+0000 7fd1ff130640 10 monclient(hunting): authenticate will time out at 7312.677663s 2024-03-25T18:15:26.456 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.455+0000 7fd1fd6cd640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fd1f81430c0 0x7fd1f814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:37684/0 (socket says 172.21.15.64:37684) 2024-03-25T18:15:26.456 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.455+0000 7fd1fd6cd640 1 -- 172.21.15.64:0/1071382347 learned_addr learned my addr 172.21.15.64:0/1071382347 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:15:26.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.469+0000 7fd1f77fe640 1 -- 172.21.15.64:0/1071382347 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259664255 0 0) 0x7fd1f80b28a0 con 0x7fd1f80a98f0 2024-03-25T18:15:26.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.469+0000 7fd1f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:26.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.469+0000 7fd1f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:26.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.469+0000 7fd1f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:26.470 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.469+0000 7fd1f77fe640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:15:26.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.469+0000 7fd1f77fe640 1 -- 172.21.15.64:0/1071382347 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e8003520 con 0x7fd1f80a98f0 2024-03-25T18:15:26.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.469+0000 7fd1f77fe640 1 -- 172.21.15.64:0/1071382347 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133303792 0 0) 0x7fd1f8142520 con 0x7fd1f81430c0 2024-03-25T18:15:26.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.470+0000 7fd1f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:26.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.470+0000 7fd1f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:26.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.470+0000 7fd1f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:26.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.470+0000 7fd1f77fe640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:15:26.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.470+0000 7fd1f77fe640 1 -- 172.21.15.64:0/1071382347 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1f80b28a0 con 0x7fd1f81430c0 2024-03-25T18:15:26.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.476+0000 7fd1f77fe640 1 -- 172.21.15.64:0/1071382347 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740326283 0 0) 0x7fd1f80b2a30 con 0x7fd1f813f710 2024-03-25T18:15:26.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.476+0000 7fd1f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:26.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.476+0000 7fd1f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:26.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.476+0000 7fd1f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:26.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.476+0000 7fd1f77fe640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:15:26.478 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.476+0000 7fd1f77fe640 1 -- 172.21.15.64:0/1071382347 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1f8142520 con 0x7fd1f813f710 2024-03-25T18:15:26.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.483+0000 7fd1f77fe640 1 -- 172.21.15.64:0/1071382347 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 433860627 0 0) 0x7fd1f80b28a0 con 0x7fd1f81430c0 2024-03-25T18:15:26.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.483+0000 7fd1f77fe640 1 -- 172.21.15.64:0/1071382347 >> v1:172.21.15.64:6790/0 conn(0x7fd1f813f710 legacy=0x7fd1f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:26.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.483+0000 7fd1f77fe640 1 -- 172.21.15.64:0/1071382347 >> v1:172.21.15.86:6789/0 conn(0x7fd1f80a98f0 legacy=0x7fd1f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:26.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.483+0000 7fd1f77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:26.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.483+0000 7fd1f77fe640 1 monclient: found mon.noname-a 2024-03-25T18:15:26.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.483+0000 7fd1f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:26.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.483+0000 7fd1f77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:15:26.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.483+0000 7fd1f77fe640 1 -- 172.21.15.64:0/1071382347 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f80b44c0 con 0x7fd1f81430c0 2024-03-25T18:15:26.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.483+0000 7fd1f77fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:26.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.483+0000 7fd1f77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:26.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.483+0000 7fd1f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:26.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.483+0000 7fd1f77fe640 1 -- 172.21.15.64:0/1071382347 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fd1f00019f0 con 0x7fd1f81430c0 2024-03-25T18:15:26.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.483+0000 7fd1ff130640 5 monclient: authenticate success, global_id 4485 2024-03-25T18:15:26.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.483+0000 7fd1f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:26.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.483+0000 7fd1f77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:26.485 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.483+0000 7fd1f77fe640 10 monclient: dump: 2024-03-25T18:15:26.485 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:26.485 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:26.485 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:26.485 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:26.485 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:26.485 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:26.485 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:26.485 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:26.485 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:26.485 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:26.485 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.483+0000 7fd1ff130640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:26.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.496+0000 7fd1f77fe640 1 -- 172.21.15.64:0/1071382347 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1f0001b50 con 0x7fd1f81430c0 2024-03-25T18:15:26.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.496+0000 7fd1f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:26.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.496+0000 7fd1f77fe640 1 -- 172.21.15.64:0/1071382347 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fd1f0001f60 con 0x7fd1f81430c0 2024-03-25T18:15:26.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.496+0000 7fd1f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:26.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.496+0000 7fd1f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:26.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.496+0000 7fd1f77fe640 10 monclient: dump: 2024-03-25T18:15:26.497 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:26.497 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:26.498 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:26.498 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:26.498 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:26.498 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:26.498 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:26.498 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:26.498 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:26.498 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:26.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.496+0000 7fd1ff130640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:26.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.496+0000 7fd1ff130640 4 set_mon_vals no callback set 2024-03-25T18:15:26.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.497+0000 7fd1ff130640 10 monclient: shutdown 2024-03-25T18:15:26.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.497+0000 7fd1ff130640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:26.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.497+0000 7fd1ff130640 1 -- 172.21.15.64:0/1071382347 >> v1:172.21.15.64:6789/0 conn(0x7fd1f81430c0 legacy=0x7fd1f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:26.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.497+0000 7fd1ff130640 1 -- 172.21.15.64:0/1071382347 shutdown_connections 2024-03-25T18:15:26.499 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.497+0000 7fd1ff130640 1 -- 172.21.15.64:0/1071382347 >> 172.21.15.64:0/1071382347 conn(0x7fd1f80b7ea0 msgr2=0x7fd1f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:26.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.497+0000 7fd1ff130640 1 -- 172.21.15.64:0/1071382347 shutdown_connections 2024-03-25T18:15:26.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.497+0000 7fd1ff130640 1 -- 172.21.15.64:0/1071382347 wait complete. 2024-03-25T18:15:26.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.498+0000 7fd1ff130640 10 monclient: build_initial_monmap 2024-03-25T18:15:26.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.498+0000 7fd1ff130640 10 monclient: monmap: 2024-03-25T18:15:26.500 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:15:26.500 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:26.500 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:15:26.500 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:15:26.500 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:26.500 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:15:26.500 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:15:26.500 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:15:26.500 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:15:26.500 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:26.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.498+0000 7fd1ff130640 1 librados: starting msgr at 2024-03-25T18:15:26.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.498+0000 7fd1ff130640 1 librados: starting objecter 2024-03-25T18:15:26.500 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.498+0000 7fd1ff130640 1 Processor -- start 2024-03-25T18:15:26.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.498+0000 7fd1ff130640 1 -- start start 2024-03-25T18:15:26.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.498+0000 7fd1ff130640 1 librados: setting wanted keys 2024-03-25T18:15:26.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.498+0000 7fd1ff130640 1 librados: calling monclient init 2024-03-25T18:15:26.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.498+0000 7fd1ff130640 10 monclient: init 2024-03-25T18:15:26.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.499+0000 7fd1ff130640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:26.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.499+0000 7fd1ff130640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:26.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.499+0000 7fd1ff130640 10 monclient(hunting): picked mon.noname-a con 0x7fd1f80a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:15:26.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.499+0000 7fd1ff130640 10 monclient(hunting): picked mon.noname-c con 0x7fd1f813f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:15:26.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.499+0000 7fd1ff130640 10 monclient(hunting): picked mon.noname-b con 0x7fd1f81430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:15:26.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.499+0000 7fd1ff130640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1f81d48f0 con 0x7fd1f80a98f0 2024-03-25T18:15:26.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.499+0000 7fd1ff130640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1f81397d0 con 0x7fd1f81430c0 2024-03-25T18:15:26.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.499+0000 7fd1ff130640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1f81d4b50 con 0x7fd1f813f710 2024-03-25T18:15:26.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.499+0000 7fd1ff130640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:26.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.499+0000 7fd1ff130640 10 monclient(hunting): authenticate will time out at 7312.738261s 2024-03-25T18:15:26.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.509+0000 7fd1fd6cd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd1f81430c0 0x7fd1f81d7380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:56650/0 (socket says 172.21.15.64:56650) 2024-03-25T18:15:26.510 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.509+0000 7fd1fd6cd640 1 -- 172.21.15.64:0/3489777653 learned_addr learned my addr 172.21.15.64:0/3489777653 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:15:26.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.510+0000 7fd1f57fa640 10 client.?.objecter ms_handle_connect 0x7fd1f81430c0 2024-03-25T18:15:26.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.510+0000 7fd1f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:26.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.515+0000 7fd1f57fa640 10 client.?.objecter ms_handle_connect 0x7fd1f813f710 2024-03-25T18:15:26.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.515+0000 7fd1f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:26.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.520+0000 7fd1f57fa640 10 client.?.objecter ms_handle_connect 0x7fd1f80a98f0 2024-03-25T18:15:26.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.520+0000 7fd1f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:26.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.522+0000 7fd1f57fa640 1 -- 172.21.15.64:0/3489777653 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1475845181 0 0) 0x7fd1f81397d0 con 0x7fd1f81430c0 2024-03-25T18:15:26.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.522+0000 7fd1f57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:26.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.522+0000 7fd1f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:26.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.522+0000 7fd1f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:26.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.522+0000 7fd1f57fa640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:15:26.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.522+0000 7fd1f57fa640 1 -- 172.21.15.64:0/3489777653 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1dc004eb0 con 0x7fd1f81430c0 2024-03-25T18:15:26.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.528+0000 7fd1f57fa640 1 -- 172.21.15.64:0/3489777653 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763132320 0 0) 0x7fd1f81d4b50 con 0x7fd1f813f710 2024-03-25T18:15:26.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.528+0000 7fd1f57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:26.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.528+0000 7fd1f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:26.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.528+0000 7fd1f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:26.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.528+0000 7fd1f57fa640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:15:26.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.528+0000 7fd1f57fa640 1 -- 172.21.15.64:0/3489777653 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1f81397d0 con 0x7fd1f813f710 2024-03-25T18:15:26.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.535+0000 7fd1f57fa640 1 -- 172.21.15.64:0/3489777653 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 203175370 0 0) 0x7fd1dc004eb0 con 0x7fd1f81430c0 2024-03-25T18:15:26.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.535+0000 7fd1f57fa640 1 -- 172.21.15.64:0/3489777653 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1f81d4b50 con 0x7fd1f81430c0 2024-03-25T18:15:26.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.535+0000 7fd1f57fa640 1 -- 172.21.15.64:0/3489777653 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fd1f0001f60 con 0x7fd1f81430c0 2024-03-25T18:15:26.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.535+0000 7fd1f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:26.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.535+0000 7fd1f57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:26.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.535+0000 7fd1f57fa640 10 monclient(hunting): dump: 2024-03-25T18:15:26.536 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:26.537 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:26.537 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:26.537 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:26.537 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:26.537 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:26.537 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:26.537 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:26.537 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:26.537 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:26.537 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.535+0000 7fd1f57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:26.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1f57fa640 1 -- 172.21.15.64:0/3489777653 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2924063698 0 0) 0x7fd1f81d4b50 con 0x7fd1f81430c0 2024-03-25T18:15:26.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1f57fa640 1 -- 172.21.15.64:0/3489777653 >> v1:172.21.15.64:6790/0 conn(0x7fd1f813f710 legacy=0x7fd1f80aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:26.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1f57fa640 1 -- 172.21.15.64:0/3489777653 >> v1:172.21.15.64:6789/0 conn(0x7fd1f80a98f0 legacy=0x7fd1f81d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:26.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1f57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:15:26.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1f57fa640 1 monclient: found mon.b 2024-03-25T18:15:26.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1f57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:26.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1f57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:26.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1f57fa640 1 -- 172.21.15.64:0/3489777653 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f80b16b0 con 0x7fd1f81430c0 2024-03-25T18:15:26.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1f57fa640 10 monclient: _finish_auth 0 2024-03-25T18:15:26.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1f57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:15:26.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:26.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1ff130640 5 monclient: authenticate success, global_id 4486 2024-03-25T18:15:26.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1ff130640 10 monclient: _renew_subs 2024-03-25T18:15:26.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1ff130640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:26.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1ff130640 1 -- 172.21.15.64:0/3489777653 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1f80b0c10 con 0x7fd1f81430c0 2024-03-25T18:15:26.547 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1ff130640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:26.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1ff130640 10 monclient: _renew_subs 2024-03-25T18:15:26.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1ff130640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:26.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1ff130640 1 -- 172.21.15.64:0/3489777653 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1f80b0fb0 con 0x7fd1f81430c0 2024-03-25T18:15:26.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.545+0000 7fd1ff130640 1 librados: init done 2024-03-25T18:15:26.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.546+0000 7fd1ff130640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:26.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.546+0000 7fd1ff130640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:26.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.546+0000 7fd1ff130640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:26.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.546+0000 7fd1ff130640 1 -- 172.21.15.64:0/3489777653 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1f81d4ce0 con 0x7fd1f81430c0 2024-03-25T18:15:26.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.814+0000 7fd1f57fa640 1 -- 172.21.15.64:0/3489777653 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1f0004e10 con 0x7fd1f81430c0 2024-03-25T18:15:26.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.814+0000 7fd1f57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:26.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.814+0000 7fd1f57fa640 1 -- 172.21.15.64:0/3489777653 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fd1f0005120 con 0x7fd1f81430c0 2024-03-25T18:15:26.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.814+0000 7fd1f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:26.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.814+0000 7fd1f57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:26.815 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.814+0000 7fd1f57fa640 10 monclient: dump: 2024-03-25T18:15:26.815 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:26.815 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:26.815 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:26.815 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:26.815 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:26.815 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:26.816 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:26.816 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:26.816 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:26.816 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:26.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.814+0000 7fd1f6ffd640 4 set_mon_vals no callback set 2024-03-25T18:15:26.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.815+0000 7fd1f57fa640 1 -- 172.21.15.64:0/3489777653 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fd1f001b0f0 con 0x7fd1f81430c0 2024-03-25T18:15:26.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.815+0000 7fd1fcecc640 10 monclient: get_auth_request con 0x7fd1dc06dd10 auth_method 0 2024-03-25T18:15:26.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.815+0000 7fd1f57fa640 10 client.4486.objecter ms_handle_connect 0x7fd1dc06dd10 2024-03-25T18:15:26.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.822+0000 7fd1f57fa640 1 -- 172.21.15.64:0/3489777653 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fd1f0054c30 con 0x7fd1f81430c0 2024-03-25T18:15:26.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.822+0000 7fd1f57fa640 10 client.4486.objecter ms_dispatch 0x7fd1f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:26.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.822+0000 7fd1f57fa640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:26.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.822+0000 7fd1f57fa640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:26.823 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.822+0000 7fd1f57fa640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-25T18:15:26.824 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.823+0000 7fd1f57fa640 1 -- 172.21.15.64:0/3489777653 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd1f0050570 con 0x7fd1f81430c0 2024-03-25T18:15:26.824 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.823+0000 7fd1f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:26.824 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:26.823+0000 7fd1f57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:27.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.060+0000 7fd1ff130640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:27.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.060+0000 7fd1ff130640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:27.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.060+0000 7fd1ff130640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:27.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.060+0000 7fd1ff130640 1 -- 172.21.15.64:0/3489777653 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1f80627b0 con 0x7fd1f81430c0 2024-03-25T18:15:27.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.088+0000 7fd1f57fa640 1 -- 172.21.15.64:0/3489777653 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fd1f0052670 con 0x7fd1f81430c0 2024-03-25T18:15:27.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.088+0000 7fd1f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:27.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.088+0000 7fd1f57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:27.089 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:15:27.089 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:27.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.091+0000 7fd1ff130640 10 librados: watch_flush enter 2024-03-25T18:15:27.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.091+0000 7fd1ff130640 10 librados: watch_flush exit 2024-03-25T18:15:27.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.091+0000 7fd1ff130640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-25T18:15:27.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.091+0000 7fd1ff130640 10 client.4486.objecter successfully canceled tick 2024-03-25T18:15:27.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.091+0000 7fd1ff130640 1 -- 172.21.15.64:0/3489777653 >> v1:172.21.15.64:6800/2915543638 conn(0x7fd1dc06dd10 legacy=0x7fd1dc0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:27.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.091+0000 7fd1ff130640 10 monclient: shutdown 2024-03-25T18:15:27.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.091+0000 7fd1ff130640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:27.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.091+0000 7fd1ff130640 1 -- 172.21.15.64:0/3489777653 >> v1:172.21.15.86:6789/0 conn(0x7fd1f81430c0 legacy=0x7fd1f81d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:27.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.091+0000 7fd1ff130640 1 -- 172.21.15.64:0/3489777653 shutdown_connections 2024-03-25T18:15:27.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.091+0000 7fd1ff130640 1 -- 172.21.15.64:0/3489777653 >> 172.21.15.64:0/3489777653 conn(0x7fd1f80b7ea0 msgr2=0x7fd1f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:27.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.091+0000 7fd1ff130640 1 -- 172.21.15.64:0/3489777653 shutdown_connections 2024-03-25T18:15:27.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.091+0000 7fd1ff130640 1 -- 172.21.15.64:0/3489777653 wait complete. 2024-03-25T18:15:27.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:27.091+0000 7fd1ff130640 1 librados: shutdown 2024-03-25T18:15:27.103 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:33.909 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:34.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.028+0000 7fe4fc693640 10 monclient: get_monmap_and_config 2024-03-25T18:15:34.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.028+0000 7fe4fc693640 10 monclient: build_initial_monmap 2024-03-25T18:15:34.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.028+0000 7fe4fc693640 10 monclient: monmap: 2024-03-25T18:15:34.029 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:15:34.029 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:34.029 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:15:34.029515+0000 2024-03-25T18:15:34.029 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:15:34.029515+0000 2024-03-25T18:15:34.030 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:34.030 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:34.030 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:15:34.030 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:15:34.030 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:15:34.030 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:34.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.029+0000 7fe4fc693640 1 Processor -- start 2024-03-25T18:15:34.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.029+0000 7fe4fc693640 1 -- start start 2024-03-25T18:15:34.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.029+0000 7fe4fc693640 10 monclient: init 2024-03-25T18:15:34.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.030+0000 7fe4fc693640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:34.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.030+0000 7fe4fc693640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:34.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.030+0000 7fe4fc693640 10 monclient(hunting): picked mon.noname-a con 0x7fe4f40a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:15:34.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.030+0000 7fe4fc693640 10 monclient(hunting): picked mon.noname-c con 0x7fe4f413f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:15:34.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.030+0000 7fe4fc693640 10 monclient(hunting): picked mon.noname-b con 0x7fe4f41430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:15:34.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.030+0000 7fe4fc693640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f4142520 con 0x7fe4f40a98f0 2024-03-25T18:15:34.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.030+0000 7fe4fc693640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f40b28a0 con 0x7fe4f413f710 2024-03-25T18:15:34.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.030+0000 7fe4fc693640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f40b2a30 con 0x7fe4f41430c0 2024-03-25T18:15:34.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.030+0000 7fe4fc693640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:34.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.030+0000 7fe4fc693640 10 monclient(hunting): authenticate will time out at 7320.268553s 2024-03-25T18:15:34.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.039+0000 7fe4fac30640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe4f41430c0 0x7fe4f414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:58664/0 (socket says 172.21.15.64:58664) 2024-03-25T18:15:34.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.039+0000 7fe4fac30640 1 -- 172.21.15.64:0/2422424870 learned_addr learned my addr 172.21.15.64:0/2422424870 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:15:34.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.048+0000 7fe4f942d640 1 -- 172.21.15.64:0/2422424870 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880175097 0 0) 0x7fe4f40b2a30 con 0x7fe4f41430c0 2024-03-25T18:15:34.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.048+0000 7fe4f942d640 10 monclient(hunting): _negotiate 2024-03-25T18:15:34.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.048+0000 7fe4f942d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:34.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.048+0000 7fe4f942d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:34.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.048+0000 7fe4f942d640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:15:34.050 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.048+0000 7fe4f942d640 1 -- 172.21.15.64:0/2422424870 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4e0003520 con 0x7fe4f41430c0 2024-03-25T18:15:34.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.054+0000 7fe4f942d640 1 -- 172.21.15.64:0/2422424870 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687327151 0 0) 0x7fe4f40b28a0 con 0x7fe4f413f710 2024-03-25T18:15:34.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.054+0000 7fe4f942d640 10 monclient(hunting): _negotiate 2024-03-25T18:15:34.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.054+0000 7fe4f942d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:34.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.054+0000 7fe4f942d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:34.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.054+0000 7fe4f942d640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:15:34.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.054+0000 7fe4f942d640 1 -- 172.21.15.64:0/2422424870 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f40b2a30 con 0x7fe4f413f710 2024-03-25T18:15:34.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.065+0000 7fe4f942d640 1 -- 172.21.15.64:0/2422424870 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2703066484 0 0) 0x7fe4f40b2a30 con 0x7fe4f413f710 2024-03-25T18:15:34.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.065+0000 7fe4f942d640 1 -- 172.21.15.64:0/2422424870 >> v1:172.21.15.64:6790/0 conn(0x7fe4f41430c0 legacy=0x7fe4f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.065+0000 7fe4f942d640 1 -- 172.21.15.64:0/2422424870 >> v1:172.21.15.64:6789/0 conn(0x7fe4f40a98f0 legacy=0x7fe4f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.065+0000 7fe4f942d640 10 monclient: _finish_hunting 0 2024-03-25T18:15:34.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.065+0000 7fe4f942d640 1 monclient: found mon.noname-c 2024-03-25T18:15:34.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.065+0000 7fe4f942d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:34.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.065+0000 7fe4f942d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:15:34.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.065+0000 7fe4f942d640 1 -- 172.21.15.64:0/2422424870 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f40b44c0 con 0x7fe4f413f710 2024-03-25T18:15:34.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.065+0000 7fe4f942d640 10 monclient: _finish_auth 0 2024-03-25T18:15:34.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.065+0000 7fe4f942d640 10 monclient: _check_auth_tickets 2024-03-25T18:15:34.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.065+0000 7fe4f942d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:34.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.065+0000 7fe4f942d640 1 -- 172.21.15.64:0/2422424870 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe4e4001670 con 0x7fe4f413f710 2024-03-25T18:15:34.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.065+0000 7fe4fc693640 5 monclient: authenticate success, global_id 4489 2024-03-25T18:15:34.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.065+0000 7fe4fc693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:34.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.065+0000 7fe4f942d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:34.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.065+0000 7fe4f942d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:34.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.065+0000 7fe4f942d640 10 monclient: dump: 2024-03-25T18:15:34.069 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:34.069 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:34.069 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:34.069 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:34.069 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:34.069 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:34.070 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:34.070 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:34.070 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:34.070 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:34.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.065+0000 7fe4fc693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:34.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.078+0000 7fe4f942d640 1 -- 172.21.15.64:0/2422424870 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4e40017d0 con 0x7fe4f413f710 2024-03-25T18:15:34.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.078+0000 7fe4f942d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:34.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.078+0000 7fe4f942d640 1 -- 172.21.15.64:0/2422424870 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe4e4001c20 con 0x7fe4f413f710 2024-03-25T18:15:34.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.079+0000 7fe4fc693640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:34.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.079+0000 7fe4fc693640 4 set_mon_vals no callback set 2024-03-25T18:15:34.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.079+0000 7fe4f942d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:34.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.079+0000 7fe4f942d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:34.080 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.079+0000 7fe4f942d640 10 monclient: dump: 2024-03-25T18:15:34.080 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:34.080 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:34.080 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:34.080 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:34.080 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:34.080 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:34.080 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:34.080 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:34.080 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:34.080 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:34.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.079+0000 7fe4fc693640 10 monclient: shutdown 2024-03-25T18:15:34.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.079+0000 7fe4fc693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:34.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.079+0000 7fe4fc693640 1 -- 172.21.15.64:0/2422424870 >> v1:172.21.15.86:6789/0 conn(0x7fe4f413f710 legacy=0x7fe4f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.079+0000 7fe4fc693640 1 -- 172.21.15.64:0/2422424870 shutdown_connections 2024-03-25T18:15:34.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.079+0000 7fe4fc693640 1 -- 172.21.15.64:0/2422424870 >> 172.21.15.64:0/2422424870 conn(0x7fe4f40b7ea0 msgr2=0x7fe4f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:34.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.079+0000 7fe4fc693640 1 -- 172.21.15.64:0/2422424870 shutdown_connections 2024-03-25T18:15:34.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.079+0000 7fe4fc693640 1 -- 172.21.15.64:0/2422424870 wait complete. 2024-03-25T18:15:34.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.080+0000 7fe4fc693640 10 monclient: build_initial_monmap 2024-03-25T18:15:34.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.080+0000 7fe4fc693640 10 monclient: monmap: 2024-03-25T18:15:34.083 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:15:34.083 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:34.083 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:15:34.083 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:15:34.083 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:34.083 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:15:34.084 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:15:34.084 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:15:34.084 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:15:34.084 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:34.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.080+0000 7fe4fc693640 1 librados: starting msgr at 2024-03-25T18:15:34.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.080+0000 7fe4fc693640 1 librados: starting objecter 2024-03-25T18:15:34.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.080+0000 7fe4fc693640 1 Processor -- start 2024-03-25T18:15:34.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.081+0000 7fe4fc693640 1 -- start start 2024-03-25T18:15:34.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.081+0000 7fe4fc693640 1 librados: setting wanted keys 2024-03-25T18:15:34.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.081+0000 7fe4fc693640 1 librados: calling monclient init 2024-03-25T18:15:34.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.081+0000 7fe4fc693640 10 monclient: init 2024-03-25T18:15:34.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.081+0000 7fe4fc693640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:34.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.081+0000 7fe4fc693640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:34.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.081+0000 7fe4fc693640 10 monclient(hunting): picked mon.noname-a con 0x7fe4f40a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:15:34.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.081+0000 7fe4fc693640 10 monclient(hunting): picked mon.noname-c con 0x7fe4f413f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:15:34.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.081+0000 7fe4fc693640 10 monclient(hunting): picked mon.noname-b con 0x7fe4f41430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:15:34.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.081+0000 7fe4fc693640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f4139520 con 0x7fe4f40a98f0 2024-03-25T18:15:34.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.081+0000 7fe4fc693640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f4139800 con 0x7fe4f41430c0 2024-03-25T18:15:34.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.081+0000 7fe4fc693640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f4134de0 con 0x7fe4f413f710 2024-03-25T18:15:34.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.081+0000 7fe4fc693640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:34.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.081+0000 7fe4fc693640 10 monclient(hunting): authenticate will time out at 7320.320189s 2024-03-25T18:15:34.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.092+0000 7fe4fac30640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe4f41430c0 0x7fe4f4133fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:56514/0 (socket says 172.21.15.64:56514) 2024-03-25T18:15:34.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.092+0000 7fe4fac30640 1 -- 172.21.15.64:0/1790052730 learned_addr learned my addr 172.21.15.64:0/1790052730 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:15:34.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.093+0000 7fe4eaffd640 10 client.?.objecter ms_handle_connect 0x7fe4f41430c0 2024-03-25T18:15:34.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.093+0000 7fe4eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:34.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.094+0000 7fe4eaffd640 10 client.?.objecter ms_handle_connect 0x7fe4f413f710 2024-03-25T18:15:34.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.094+0000 7fe4eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:34.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.095+0000 7fe4eaffd640 10 client.?.objecter ms_handle_connect 0x7fe4f40a98f0 2024-03-25T18:15:34.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.095+0000 7fe4eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:34.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.101+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998950452 0 0) 0x7fe4f4139800 con 0x7fe4f41430c0 2024-03-25T18:15:34.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.101+0000 7fe4eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:34.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.101+0000 7fe4eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.101+0000 7fe4eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.101+0000 7fe4eaffd640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:15:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.101+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d8004eb0 con 0x7fe4f41430c0 2024-03-25T18:15:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.102+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493689802 0 0) 0x7fe4f4134de0 con 0x7fe4f413f710 2024-03-25T18:15:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.102+0000 7fe4eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.102+0000 7fe4eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.102+0000 7fe4eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.102+0000 7fe4eaffd640 10 monclient(hunting): my global_id is 4499 2024-03-25T18:15:34.103 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.102+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f4139800 con 0x7fe4f413f710 2024-03-25T18:15:34.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.124+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240764536 0 0) 0x7fe4f4139520 con 0x7fe4f40a98f0 2024-03-25T18:15:34.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.124+0000 7fe4eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:34.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.124+0000 7fe4eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:34.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.124+0000 7fe4eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:34.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.124+0000 7fe4eaffd640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:15:34.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.124+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f4134de0 con 0x7fe4f40a98f0 2024-03-25T18:15:34.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.124+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2752143220 0 0) 0x7fe4d8004eb0 con 0x7fe4f41430c0 2024-03-25T18:15:34.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.124+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4f4139520 con 0x7fe4f41430c0 2024-03-25T18:15:34.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.124+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe4ec001b30 con 0x7fe4f41430c0 2024-03-25T18:15:34.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.124+0000 7fe4eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:34.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.124+0000 7fe4eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:34.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.124+0000 7fe4eaffd640 10 monclient(hunting): dump: 2024-03-25T18:15:34.126 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:34.126 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:34.126 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:34.126 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:34.126 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:34.126 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:34.127 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:34.127 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:34.127 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:34.127 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:34.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.124+0000 7fe4eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:34.130 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.129+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 44400297 0 0) 0x7fe4f4139800 con 0x7fe4f413f710 2024-03-25T18:15:34.130 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.129+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4d8004eb0 con 0x7fe4f413f710 2024-03-25T18:15:34.130 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.129+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe4e4001b20 con 0x7fe4f413f710 2024-03-25T18:15:34.130 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.129+0000 7fe4eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:34.130 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.129+0000 7fe4eaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:34.130 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.129+0000 7fe4eaffd640 10 monclient(hunting): dump: 2024-03-25T18:15:34.130 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:34.130 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:34.131 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:34.131 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:34.131 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:34.131 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:34.131 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:34.131 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:34.131 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:34.131 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:34.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 877215361 0 0) 0x7fe4f4139520 con 0x7fe4f41430c0 2024-03-25T18:15:34.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 >> v1:172.21.15.64:6790/0 conn(0x7fe4f413f710 legacy=0x7fe4f41d77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 >> v1:172.21.15.64:6789/0 conn(0x7fe4f40a98f0 legacy=0x7fe4f4138470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4eaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:34.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4eaffd640 1 monclient: found mon.b 2024-03-25T18:15:34.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:34.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4eaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:34.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f4139080 con 0x7fe4f41430c0 2024-03-25T18:15:34.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4eaffd640 10 monclient: _finish_auth 0 2024-03-25T18:15:34.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4eaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:34.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:34.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4fc693640 5 monclient: authenticate success, global_id 4492 2024-03-25T18:15:34.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4fc693640 10 monclient: _renew_subs 2024-03-25T18:15:34.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4fc693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:34.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4fc693640 1 -- 172.21.15.64:0/1790052730 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f4138b80 con 0x7fe4f41430c0 2024-03-25T18:15:34.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4fc693640 10 client.4492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:34.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4fc693640 10 monclient: _renew_subs 2024-03-25T18:15:34.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4fc693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:34.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4fc693640 1 -- 172.21.15.64:0/1790052730 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f41d80c0 con 0x7fe4f41430c0 2024-03-25T18:15:34.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.137+0000 7fe4fc693640 1 librados: init done 2024-03-25T18:15:34.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.138+0000 7fe4fc693640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:34.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.138+0000 7fe4fc693640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:34.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.138+0000 7fe4fc693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:34.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.138+0000 7fe4fc693640 1 -- 172.21.15.64:0/1790052730 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f4139290 con 0x7fe4f41430c0 2024-03-25T18:15:34.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.435+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4ec001fe0 con 0x7fe4f41430c0 2024-03-25T18:15:34.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.435+0000 7fe4eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:34.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.435+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe4ec004d00 con 0x7fe4f41430c0 2024-03-25T18:15:34.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.435+0000 7fe4eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:34.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.435+0000 7fe4eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:34.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.435+0000 7fe4eaffd640 10 monclient: dump: 2024-03-25T18:15:34.436 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:34.436 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:34.436 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.435+0000 7fe4f8c2c640 4 set_mon_vals no callback set 2024-03-25T18:15:34.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.435+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fe4ec01acd0 con 0x7fe4f41430c0 2024-03-25T18:15:34.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.436+0000 7fe4fa42f640 10 monclient: get_auth_request con 0x7fe4d806ded0 auth_method 0 2024-03-25T18:15:34.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.436+0000 7fe4eaffd640 10 client.4492.objecter ms_handle_connect 0x7fe4d806ded0 2024-03-25T18:15:34.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.440+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fe4ec054af0 con 0x7fe4f41430c0 2024-03-25T18:15:34.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.440+0000 7fe4eaffd640 10 client.4492.objecter ms_dispatch 0x7fe4f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:34.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.440+0000 7fe4eaffd640 3 client.4492.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:34.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.440+0000 7fe4eaffd640 3 client.4492.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:34.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.440+0000 7fe4eaffd640 20 client.4492.objecter dump_active .. 0 homeless 2024-03-25T18:15:34.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.440+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4ec050430 con 0x7fe4f41430c0 2024-03-25T18:15:34.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.440+0000 7fe4eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:34.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.440+0000 7fe4eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:34.689 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.688+0000 7fe4fc693640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:34.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.688+0000 7fe4fc693640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:34.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.688+0000 7fe4fc693640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:34.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.688+0000 7fe4fc693640 1 -- 172.21.15.64:0/1790052730 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4f40627b0 con 0x7fe4f41430c0 2024-03-25T18:15:34.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.722+0000 7fe4eaffd640 1 -- 172.21.15.64:0/1790052730 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fe4ec052530 con 0x7fe4f41430c0 2024-03-25T18:15:34.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.722+0000 7fe4eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:34.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.722+0000 7fe4eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:34.724 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:15:34.724 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:34.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.725+0000 7fe4fc693640 10 librados: watch_flush enter 2024-03-25T18:15:34.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.725+0000 7fe4fc693640 10 librados: watch_flush exit 2024-03-25T18:15:34.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.725+0000 7fe4fc693640 20 client.4492.objecter shutdown clearing up homeless session... 2024-03-25T18:15:34.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.725+0000 7fe4fc693640 10 client.4492.objecter successfully canceled tick 2024-03-25T18:15:34.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.725+0000 7fe4fc693640 1 -- 172.21.15.64:0/1790052730 >> v1:172.21.15.64:6800/2915543638 conn(0x7fe4d806ded0 legacy=0x7fe4d8070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.725+0000 7fe4fc693640 10 monclient: shutdown 2024-03-25T18:15:34.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.725+0000 7fe4fc693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:34.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.725+0000 7fe4fc693640 1 -- 172.21.15.64:0/1790052730 >> v1:172.21.15.86:6789/0 conn(0x7fe4f41430c0 legacy=0x7fe4f4133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.725+0000 7fe4fc693640 1 -- 172.21.15.64:0/1790052730 shutdown_connections 2024-03-25T18:15:34.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.725+0000 7fe4fc693640 1 -- 172.21.15.64:0/1790052730 >> 172.21.15.64:0/1790052730 conn(0x7fe4f40b7ea0 msgr2=0x7fe4f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:34.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.725+0000 7fe4fc693640 1 -- 172.21.15.64:0/1790052730 shutdown_connections 2024-03-25T18:15:34.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.725+0000 7fe4fc693640 1 -- 172.21.15.64:0/1790052730 wait complete. 2024-03-25T18:15:34.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:34.725+0000 7fe4fc693640 1 librados: shutdown 2024-03-25T18:15:34.738 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:37.529 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:06:20.641 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:37.534 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:06:20.647 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:37.581 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:06:20.693 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:37.588 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:06:20.701 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:37.591 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:06:20.670 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:37.599 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:06:20.712 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:37.601 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:06:20.679 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:37.629 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:06:20.742 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:37.641 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:06:20.720 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:41.545 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:41.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.657+0000 7f78cd1cf640 10 monclient: get_monmap_and_config 2024-03-25T18:15:41.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.657+0000 7f78cd1cf640 10 monclient: build_initial_monmap 2024-03-25T18:15:41.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.657+0000 7f78cd1cf640 10 monclient: monmap: 2024-03-25T18:15:41.659 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:15:41.659 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:41.659 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:15:41.658369+0000 2024-03-25T18:15:41.659 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:15:41.658369+0000 2024-03-25T18:15:41.659 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:41.659 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:41.659 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:15:41.659 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:15:41.660 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:15:41.660 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:41.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.658+0000 7f78cd1cf640 1 Processor -- start 2024-03-25T18:15:41.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.658+0000 7f78cd1cf640 1 -- start start 2024-03-25T18:15:41.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.658+0000 7f78cd1cf640 10 monclient: init 2024-03-25T18:15:41.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.659+0000 7f78cd1cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:41.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.659+0000 7f78cd1cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:41.661 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.659+0000 7f78cd1cf640 10 monclient(hunting): picked mon.noname-c con 0x7f78c80b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:15:41.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.659+0000 7f78cd1cf640 10 monclient(hunting): picked mon.noname-a con 0x7f78c80b1730 addr v1:172.21.15.64:6789/0 2024-03-25T18:15:41.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.659+0000 7f78cd1cf640 10 monclient(hunting): picked mon.noname-b con 0x7f78c814be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:15:41.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.659+0000 7f78cd1cf640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c80af940 con 0x7f78c80b1730 2024-03-25T18:15:41.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.659+0000 7f78cd1cf640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c80b0a70 con 0x7f78c80b4d60 2024-03-25T18:15:41.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.659+0000 7f78cd1cf640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c80b0d50 con 0x7f78c814be90 2024-03-25T18:15:41.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.659+0000 7f78cd1cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:41.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.659+0000 7f78cd1cf640 10 monclient(hunting): authenticate will time out at 7327.898175s 2024-03-25T18:15:41.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.673+0000 7f78c659c640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f78c80b1730 0x7f78c80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:56994/0 (socket says 172.21.15.64:56994) 2024-03-25T18:15:41.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.673+0000 7f78c6d9d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f78c80b4d60 0x7f78c813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:35476/0 (socket says 172.21.15.64:35476) 2024-03-25T18:15:41.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.673+0000 7f78c659c640 1 -- 172.21.15.64:0/2086422905 learned_addr learned my addr 172.21.15.64:0/2086422905 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:15:41.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.677+0000 7f78c5d9b640 1 -- 172.21.15.64:0/2086422905 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450500586 0 0) 0x7f78c80b0a70 con 0x7f78c80b4d60 2024-03-25T18:15:41.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.677+0000 7f78c5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:41.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.677+0000 7f78c5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:41.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.677+0000 7f78c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:41.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.677+0000 7f78c5d9b640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:15:41.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.677+0000 7f78c5d9b640 1 -- 172.21.15.64:0/2086422905 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78b0003520 con 0x7f78c80b4d60 2024-03-25T18:15:41.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.691+0000 7f78c5d9b640 1 -- 172.21.15.64:0/2086422905 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2285251398 0 0) 0x7f78b0003520 con 0x7f78c80b4d60 2024-03-25T18:15:41.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.692+0000 7f78c5d9b640 1 -- 172.21.15.64:0/2086422905 >> v1:172.21.15.64:6790/0 conn(0x7f78c814be90 legacy=0x7f78c814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:41.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.692+0000 7f78c5d9b640 1 -- 172.21.15.64:0/2086422905 >> v1:172.21.15.64:6789/0 conn(0x7f78c80b1730 legacy=0x7f78c80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:41.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.692+0000 7f78c5d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:15:41.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.692+0000 7f78c5d9b640 1 monclient: found mon.noname-c 2024-03-25T18:15:41.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.692+0000 7f78c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:41.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.692+0000 7f78c5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:15:41.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.692+0000 7f78c5d9b640 1 -- 172.21.15.64:0/2086422905 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c80b1030 con 0x7f78c80b4d60 2024-03-25T18:15:41.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.692+0000 7f78c5d9b640 10 monclient: _finish_auth 0 2024-03-25T18:15:41.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.692+0000 7f78c5d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:15:41.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.692+0000 7f78c5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:41.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.692+0000 7f78c5d9b640 1 -- 172.21.15.64:0/2086422905 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f78b8001690 con 0x7f78c80b4d60 2024-03-25T18:15:41.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.692+0000 7f78c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:41.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.692+0000 7f78c5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:41.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.692+0000 7f78c5d9b640 10 monclient: dump: 2024-03-25T18:15:41.694 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:41.694 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:41.694 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:41.694 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:41.694 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:41.694 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:41.694 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:41.694 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:41.694 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:41.695 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:41.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.692+0000 7f78cd1cf640 5 monclient: authenticate success, global_id 4495 2024-03-25T18:15:41.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.692+0000 7f78cd1cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:41.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.721+0000 7f78c5d9b640 1 -- 172.21.15.64:0/2086422905 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78b80017f0 con 0x7f78c80b4d60 2024-03-25T18:15:41.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.721+0000 7f78c5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:41.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.721+0000 7f78c5d9b640 1 -- 172.21.15.64:0/2086422905 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f78b8001c00 con 0x7f78c80b4d60 2024-03-25T18:15:41.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.721+0000 7f78c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:41.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.721+0000 7f78c5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:41.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.721+0000 7f78c5d9b640 10 monclient: dump: 2024-03-25T18:15:41.723 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:41.723 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:41.723 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:41.723 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:41.723 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:41.723 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:41.723 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:41.723 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:41.723 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:41.723 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:41.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.721+0000 7f78cd1cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:41.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.721+0000 7f78cd1cf640 4 set_mon_vals no callback set 2024-03-25T18:15:41.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.722+0000 7f78cd1cf640 10 monclient: shutdown 2024-03-25T18:15:41.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.722+0000 7f78cd1cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:41.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.722+0000 7f78cd1cf640 1 -- 172.21.15.64:0/2086422905 >> v1:172.21.15.86:6789/0 conn(0x7f78c80b4d60 legacy=0x7f78c813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:41.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.722+0000 7f78cd1cf640 1 -- 172.21.15.64:0/2086422905 shutdown_connections 2024-03-25T18:15:41.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.722+0000 7f78cd1cf640 1 -- 172.21.15.64:0/2086422905 >> 172.21.15.64:0/2086422905 conn(0x7f78c80617a0 msgr2=0x7f78c8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:41.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.723+0000 7f78cd1cf640 1 -- 172.21.15.64:0/2086422905 shutdown_connections 2024-03-25T18:15:41.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.723+0000 7f78cd1cf640 1 -- 172.21.15.64:0/2086422905 wait complete. 2024-03-25T18:15:41.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.724+0000 7f78cd1cf640 10 monclient: build_initial_monmap 2024-03-25T18:15:41.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.724+0000 7f78cd1cf640 10 monclient: monmap: 2024-03-25T18:15:41.725 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:15:41.725 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:41.725 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:15:41.725 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:15:41.725 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:41.725 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:15:41.725 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:15:41.725 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:15:41.725 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:15:41.725 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:41.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.724+0000 7f78cd1cf640 1 librados: starting msgr at 2024-03-25T18:15:41.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.724+0000 7f78cd1cf640 1 librados: starting objecter 2024-03-25T18:15:41.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.724+0000 7f78cd1cf640 1 Processor -- start 2024-03-25T18:15:41.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.725+0000 7f78cd1cf640 1 -- start start 2024-03-25T18:15:41.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.725+0000 7f78cd1cf640 1 librados: setting wanted keys 2024-03-25T18:15:41.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.725+0000 7f78cd1cf640 1 librados: calling monclient init 2024-03-25T18:15:41.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.725+0000 7f78cd1cf640 10 monclient: init 2024-03-25T18:15:41.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.725+0000 7f78cd1cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:41.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.725+0000 7f78cd1cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:41.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.725+0000 7f78cd1cf640 10 monclient(hunting): picked mon.noname-b con 0x7f78c80b1730 addr v1:172.21.15.86:6789/0 2024-03-25T18:15:41.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.725+0000 7f78cd1cf640 10 monclient(hunting): picked mon.noname-c con 0x7f78c80b4d60 addr v1:172.21.15.64:6790/0 2024-03-25T18:15:41.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.725+0000 7f78cd1cf640 10 monclient(hunting): picked mon.noname-a con 0x7f78c814be90 addr v1:172.21.15.64:6789/0 2024-03-25T18:15:41.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.725+0000 7f78cd1cf640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c8142050 con 0x7f78c814be90 2024-03-25T18:15:41.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.725+0000 7f78cd1cf640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c81dac50 con 0x7f78c80b1730 2024-03-25T18:15:41.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.726+0000 7f78cd1cf640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c8140d00 con 0x7f78c80b4d60 2024-03-25T18:15:41.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.726+0000 7f78cd1cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:41.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.726+0000 7f78cd1cf640 10 monclient(hunting): authenticate will time out at 7327.964563s 2024-03-25T18:15:41.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.735+0000 7f78c759e640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f78c814be90 0x7f78c81e2260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:57048/0 (socket says 172.21.15.64:57048) 2024-03-25T18:15:41.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.735+0000 7f78c759e640 1 -- 172.21.15.64:0/120181046 learned_addr learned my addr 172.21.15.64:0/120181046 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:15:41.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.742+0000 7f78af7fe640 10 client.?.objecter ms_handle_connect 0x7f78c80b1730 2024-03-25T18:15:41.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.742+0000 7f78af7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:41.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.742+0000 7f78af7fe640 10 client.?.objecter ms_handle_connect 0x7f78c80b4d60 2024-03-25T18:15:41.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.742+0000 7f78af7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:41.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.747+0000 7f78af7fe640 10 client.?.objecter ms_handle_connect 0x7f78c814be90 2024-03-25T18:15:41.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.747+0000 7f78af7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:41.750 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.749+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563730955 0 0) 0x7f78c81dac50 con 0x7f78c80b1730 2024-03-25T18:15:41.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.749+0000 7f78af7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:41.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.749+0000 7f78af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:41.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.749+0000 7f78af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:41.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.749+0000 7f78af7fe640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:15:41.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.749+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78b4004eb0 con 0x7f78c80b1730 2024-03-25T18:15:41.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.755+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3604746319 0 0) 0x7f78c8140d00 con 0x7f78c80b4d60 2024-03-25T18:15:41.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.755+0000 7f78af7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:41.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.755+0000 7f78af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:41.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.755+0000 7f78af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:41.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.755+0000 7f78af7fe640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:15:41.756 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.755+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78c81dac50 con 0x7f78c80b4d60 2024-03-25T18:15:41.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.766+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402460353 0 0) 0x7f78c8142050 con 0x7f78c814be90 2024-03-25T18:15:41.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.766+0000 7f78af7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:41.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.766+0000 7f78af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:41.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.766+0000 7f78af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:41.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.766+0000 7f78af7fe640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:15:41.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.766+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78c8140d00 con 0x7f78c814be90 2024-03-25T18:15:41.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.772+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 20730468 0 0) 0x7f78b4004eb0 con 0x7f78c80b1730 2024-03-25T18:15:41.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.772+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78c8142050 con 0x7f78c80b1730 2024-03-25T18:15:41.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.772+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f78b8001b00 con 0x7f78c80b1730 2024-03-25T18:15:41.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.772+0000 7f78af7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:41.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.772+0000 7f78af7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:41.773 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.772+0000 7f78af7fe640 10 monclient(hunting): dump: 2024-03-25T18:15:41.773 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:41.773 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:41.773 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:41.773 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:41.773 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:41.773 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:41.773 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:41.774 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:41.774 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:41.774 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:41.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.772+0000 7f78af7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:41.779 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.778+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 898506192 0 0) 0x7f78c8140d00 con 0x7f78c814be90 2024-03-25T18:15:41.779 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.778+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78b4004eb0 con 0x7f78c814be90 2024-03-25T18:15:41.779 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.778+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f78c0001bd0 con 0x7f78c814be90 2024-03-25T18:15:41.779 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.778+0000 7f78af7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:41.779 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.778+0000 7f78af7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:41.779 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.778+0000 7f78af7fe640 10 monclient(hunting): dump: 2024-03-25T18:15:41.779 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:41.779 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:41.779 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:41.779 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:41.779 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:41.779 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:41.779 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:41.779 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:41.780 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:41.780 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:41.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.780+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 486679644 0 0) 0x7f78c81dac50 con 0x7f78c80b4d60 2024-03-25T18:15:41.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.780+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78c8140d00 con 0x7f78c80b4d60 2024-03-25T18:15:41.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.780+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f78bc0018b0 con 0x7f78c80b4d60 2024-03-25T18:15:41.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.780+0000 7f78af7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:41.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.780+0000 7f78af7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:41.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.780+0000 7f78af7fe640 10 monclient(hunting): dump: 2024-03-25T18:15:41.781 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:41.781 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:41.781 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:41.781 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:41.781 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:41.781 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:41.781 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:41.782 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:41.782 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:41.782 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:41.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.780+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3667055788 0 0) 0x7f78c8142050 con 0x7f78c80b1730 2024-03-25T18:15:41.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.780+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 >> v1:172.21.15.64:6790/0 conn(0x7f78c80b4d60 legacy=0x7f78c81db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:41.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.780+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 >> v1:172.21.15.64:6789/0 conn(0x7f78c814be90 legacy=0x7f78c81e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:41.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.780+0000 7f78af7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:41.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.780+0000 7f78af7fe640 1 monclient: found mon.b 2024-03-25T18:15:41.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.780+0000 7f78af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:41.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.780+0000 7f78af7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:41.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.780+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c81423e0 con 0x7f78c80b1730 2024-03-25T18:15:41.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.780+0000 7f78af7fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:41.783 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.780+0000 7f78af7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:41.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.780+0000 7f78af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:41.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.781+0000 7f78cd1cf640 5 monclient: authenticate success, global_id 4498 2024-03-25T18:15:41.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.781+0000 7f78cd1cf640 10 monclient: _renew_subs 2024-03-25T18:15:41.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.781+0000 7f78cd1cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:41.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.781+0000 7f78cd1cf640 1 -- 172.21.15.64:0/120181046 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c813b470 con 0x7f78c80b1730 2024-03-25T18:15:41.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.781+0000 7f78cd1cf640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:41.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.781+0000 7f78cd1cf640 10 monclient: _renew_subs 2024-03-25T18:15:41.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.781+0000 7f78cd1cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:41.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.781+0000 7f78cd1cf640 1 -- 172.21.15.64:0/120181046 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c81dba40 con 0x7f78c80b1730 2024-03-25T18:15:41.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.781+0000 7f78cd1cf640 1 librados: init done 2024-03-25T18:15:41.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.781+0000 7f78cd1cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:41.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.781+0000 7f78cd1cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:41.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.781+0000 7f78cd1cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:41.784 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:41.781+0000 7f78cd1cf640 1 -- 172.21.15.64:0/120181046 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78c81401f0 con 0x7f78c80b1730 2024-03-25T18:15:42.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.062+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78b8004a90 con 0x7f78c80b1730 2024-03-25T18:15:42.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.062+0000 7f78af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:42.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.062+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f78b8004da0 con 0x7f78c80b1730 2024-03-25T18:15:42.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.062+0000 7f78af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:42.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.062+0000 7f78af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:42.064 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.062+0000 7f78af7fe640 10 monclient: dump: 2024-03-25T18:15:42.064 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:42.064 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:42.064 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:42.064 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:42.064 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:42.064 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:42.064 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:42.064 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:42.065 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:42.065 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:42.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.062+0000 7f78c559a640 4 set_mon_vals no callback set 2024-03-25T18:15:42.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.063+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f78b801ad70 con 0x7f78c80b1730 2024-03-25T18:15:42.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.064+0000 7f78c659c640 10 monclient: get_auth_request con 0x7f78b406e0f0 auth_method 0 2024-03-25T18:15:42.066 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.064+0000 7f78af7fe640 10 client.4498.objecter ms_handle_connect 0x7f78b406e0f0 2024-03-25T18:15:42.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.067+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f78b8054800 con 0x7f78c80b1730 2024-03-25T18:15:42.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.067+0000 7f78af7fe640 10 client.4498.objecter ms_dispatch 0x7f78c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:42.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.067+0000 7f78af7fe640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:42.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.067+0000 7f78af7fe640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:42.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.067+0000 7f78af7fe640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-25T18:15:42.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.067+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f78b8050310 con 0x7f78c80b1730 2024-03-25T18:15:42.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.067+0000 7f78af7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:42.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.067+0000 7f78af7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:42.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.298+0000 7f78cd1cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:42.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.298+0000 7f78cd1cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:42.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.298+0000 7f78cd1cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:42.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.298+0000 7f78cd1cf640 1 -- 172.21.15.64:0/120181046 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78c8140e90 con 0x7f78c80b1730 2024-03-25T18:15:42.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.330+0000 7f78af7fe640 1 -- 172.21.15.64:0/120181046 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f78b8052240 con 0x7f78c80b1730 2024-03-25T18:15:42.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.330+0000 7f78af7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:42.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.330+0000 7f78af7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:42.332 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:15:42.332 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:42.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.332+0000 7f78cd1cf640 10 librados: watch_flush enter 2024-03-25T18:15:42.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.332+0000 7f78cd1cf640 10 librados: watch_flush exit 2024-03-25T18:15:42.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.332+0000 7f78cd1cf640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-25T18:15:42.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.332+0000 7f78cd1cf640 10 client.4498.objecter successfully canceled tick 2024-03-25T18:15:42.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.332+0000 7f78cd1cf640 1 -- 172.21.15.64:0/120181046 >> v1:172.21.15.64:6800/2915543638 conn(0x7f78b406e0f0 legacy=0x7f78b40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:42.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.332+0000 7f78cd1cf640 10 monclient: shutdown 2024-03-25T18:15:42.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.332+0000 7f78cd1cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:42.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.332+0000 7f78cd1cf640 1 -- 172.21.15.64:0/120181046 >> v1:172.21.15.86:6789/0 conn(0x7f78c80b1730 legacy=0x7f78c81405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:42.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.332+0000 7f78cd1cf640 1 -- 172.21.15.64:0/120181046 shutdown_connections 2024-03-25T18:15:42.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.332+0000 7f78cd1cf640 1 -- 172.21.15.64:0/120181046 >> 172.21.15.64:0/120181046 conn(0x7f78c80617a0 msgr2=0x7f78c80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:42.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.332+0000 7f78cd1cf640 1 -- 172.21.15.64:0/120181046 shutdown_connections 2024-03-25T18:15:42.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.332+0000 7f78cd1cf640 1 -- 172.21.15.64:0/120181046 wait complete. 2024-03-25T18:15:42.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:42.332+0000 7f78cd1cf640 1 librados: shutdown 2024-03-25T18:15:42.346 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:49.151 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:49.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.269+0000 7fbaf2bba640 10 monclient: get_monmap_and_config 2024-03-25T18:15:49.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.269+0000 7fbaf2bba640 10 monclient: build_initial_monmap 2024-03-25T18:15:49.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.269+0000 7fbaf2bba640 10 monclient: monmap: 2024-03-25T18:15:49.270 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:15:49.270 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:49.270 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:15:49.269949+0000 2024-03-25T18:15:49.270 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:15:49.269949+0000 2024-03-25T18:15:49.270 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:49.270 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:49.270 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:15:49.270 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:15:49.270 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:15:49.271 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:49.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.269+0000 7fbaf2bba640 1 Processor -- start 2024-03-25T18:15:49.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.269+0000 7fbaf2bba640 1 -- start start 2024-03-25T18:15:49.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.269+0000 7fbaf2bba640 10 monclient: init 2024-03-25T18:15:49.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.270+0000 7fbaf2bba640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:49.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.270+0000 7fbaf2bba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:49.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.270+0000 7fbaf2bba640 10 monclient(hunting): picked mon.noname-a con 0x7fbaec0a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:15:49.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.270+0000 7fbaf2bba640 10 monclient(hunting): picked mon.noname-c con 0x7fbaec13f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:15:49.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.270+0000 7fbaf2bba640 10 monclient(hunting): picked mon.noname-b con 0x7fbaec1430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:15:49.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.270+0000 7fbaf2bba640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaec142520 con 0x7fbaec0a98f0 2024-03-25T18:15:49.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.270+0000 7fbaf2bba640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaec0b28a0 con 0x7fbaec13f710 2024-03-25T18:15:49.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.270+0000 7fbaf2bba640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaec0b2a30 con 0x7fbaec1430c0 2024-03-25T18:15:49.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.270+0000 7fbaf2bba640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:49.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.270+0000 7fbaf2bba640 10 monclient(hunting): authenticate will time out at 7335.509153s 2024-03-25T18:15:49.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.281+0000 7fbaebfff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fbaec13f710 0x7fbaec1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:49020/0 (socket says 172.21.15.64:49020) 2024-03-25T18:15:49.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.281+0000 7fbaebfff640 1 -- 172.21.15.64:0/2330986969 learned_addr learned my addr 172.21.15.64:0/2330986969 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:15:49.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.290+0000 7fbaeb7fe640 1 -- 172.21.15.64:0/2330986969 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140218501 0 0) 0x7fbaec0b28a0 con 0x7fbaec13f710 2024-03-25T18:15:49.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.290+0000 7fbaeb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:49.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.290+0000 7fbaeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:49.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.290+0000 7fbaeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:49.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.290+0000 7fbaeb7fe640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:15:49.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.290+0000 7fbaeb7fe640 1 -- 172.21.15.64:0/2330986969 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbadc003520 con 0x7fbaec13f710 2024-03-25T18:15:49.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.303+0000 7fbaeb7fe640 1 -- 172.21.15.64:0/2330986969 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1405081956 0 0) 0x7fbadc003520 con 0x7fbaec13f710 2024-03-25T18:15:49.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.303+0000 7fbaeb7fe640 1 -- 172.21.15.64:0/2330986969 >> v1:172.21.15.64:6790/0 conn(0x7fbaec1430c0 legacy=0x7fbaec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:49.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.303+0000 7fbaeb7fe640 1 -- 172.21.15.64:0/2330986969 >> v1:172.21.15.64:6789/0 conn(0x7fbaec0a98f0 legacy=0x7fbaec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:49.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.303+0000 7fbaeb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:49.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.303+0000 7fbaeb7fe640 1 monclient: found mon.noname-c 2024-03-25T18:15:49.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.303+0000 7fbaeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:49.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.303+0000 7fbaeb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:15:49.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.303+0000 7fbaeb7fe640 1 -- 172.21.15.64:0/2330986969 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaec0b44c0 con 0x7fbaec13f710 2024-03-25T18:15:49.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.303+0000 7fbaeb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:49.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.303+0000 7fbaeb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:49.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.303+0000 7fbaeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:49.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.303+0000 7fbaeb7fe640 1 -- 172.21.15.64:0/2330986969 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbad8001690 con 0x7fbaec13f710 2024-03-25T18:15:49.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.303+0000 7fbaeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:49.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.303+0000 7fbaeb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:49.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.303+0000 7fbaeb7fe640 10 monclient: dump: 2024-03-25T18:15:49.305 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:49.305 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:49.305 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:49.305 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:49.305 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:49.305 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:49.306 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:49.306 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:49.306 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:49.306 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:49.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.303+0000 7fbaf2bba640 5 monclient: authenticate success, global_id 4501 2024-03-25T18:15:49.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.303+0000 7fbaf2bba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:49.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.324+0000 7fbaeb7fe640 1 -- 172.21.15.64:0/2330986969 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbad80017f0 con 0x7fbaec13f710 2024-03-25T18:15:49.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.324+0000 7fbaeb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:49.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.324+0000 7fbaeb7fe640 1 -- 172.21.15.64:0/2330986969 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbad8001c00 con 0x7fbaec13f710 2024-03-25T18:15:49.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.324+0000 7fbaeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:49.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.324+0000 7fbaeb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:49.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.324+0000 7fbaeb7fe640 10 monclient: dump: 2024-03-25T18:15:49.326 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:49.326 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:49.326 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:49.326 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:49.326 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:49.326 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:49.326 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:49.326 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:49.326 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:49.326 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:49.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.324+0000 7fbaf2bba640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:49.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.324+0000 7fbaf2bba640 4 set_mon_vals no callback set 2024-03-25T18:15:49.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.325+0000 7fbaf2bba640 10 monclient: shutdown 2024-03-25T18:15:49.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.325+0000 7fbaf2bba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:49.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.325+0000 7fbaf2bba640 1 -- 172.21.15.64:0/2330986969 >> v1:172.21.15.86:6789/0 conn(0x7fbaec13f710 legacy=0x7fbaec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:49.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.326+0000 7fbaf2bba640 1 -- 172.21.15.64:0/2330986969 shutdown_connections 2024-03-25T18:15:49.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.326+0000 7fbaf2bba640 1 -- 172.21.15.64:0/2330986969 >> 172.21.15.64:0/2330986969 conn(0x7fbaec0b7ea0 msgr2=0x7fbaec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:49.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.326+0000 7fbaf2bba640 1 -- 172.21.15.64:0/2330986969 shutdown_connections 2024-03-25T18:15:49.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.326+0000 7fbaf2bba640 1 -- 172.21.15.64:0/2330986969 wait complete. 2024-03-25T18:15:49.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.326+0000 7fbaf2bba640 10 monclient: build_initial_monmap 2024-03-25T18:15:49.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.326+0000 7fbaf2bba640 10 monclient: monmap: 2024-03-25T18:15:49.328 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:15:49.328 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:49.328 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:15:49.329 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:15:49.329 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:49.329 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:15:49.329 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:15:49.329 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:15:49.329 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:15:49.329 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:49.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.327+0000 7fbaf2bba640 1 librados: starting msgr at 2024-03-25T18:15:49.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.327+0000 7fbaf2bba640 1 librados: starting objecter 2024-03-25T18:15:49.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.327+0000 7fbaf2bba640 1 Processor -- start 2024-03-25T18:15:49.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.327+0000 7fbaf2bba640 1 -- start start 2024-03-25T18:15:49.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.327+0000 7fbaf2bba640 1 librados: setting wanted keys 2024-03-25T18:15:49.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.327+0000 7fbaf2bba640 1 librados: calling monclient init 2024-03-25T18:15:49.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.327+0000 7fbaf2bba640 10 monclient: init 2024-03-25T18:15:49.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.328+0000 7fbaf2bba640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:49.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.328+0000 7fbaf2bba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:49.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.328+0000 7fbaf2bba640 10 monclient(hunting): picked mon.noname-c con 0x7fbaec0a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:15:49.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.328+0000 7fbaf2bba640 10 monclient(hunting): picked mon.noname-a con 0x7fbaec13f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:15:49.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.328+0000 7fbaf2bba640 10 monclient(hunting): picked mon.noname-b con 0x7fbaec1430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:15:49.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.328+0000 7fbaf2bba640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaec0b1870 con 0x7fbaec13f710 2024-03-25T18:15:49.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.328+0000 7fbaf2bba640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaec0ac910 con 0x7fbaec1430c0 2024-03-25T18:15:49.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.329+0000 7fbaf2bba640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaec0abb00 con 0x7fbaec0a98f0 2024-03-25T18:15:49.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.329+0000 7fbaf2bba640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:49.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.329+0000 7fbaf2bba640 10 monclient(hunting): authenticate will time out at 7335.567710s 2024-03-25T18:15:49.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.335+0000 7fbaf0956640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fbaec0a98f0 0x7fbaec138670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:58056/0 (socket says 172.21.15.64:58056) 2024-03-25T18:15:49.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.335+0000 7fbaf0956640 1 -- 172.21.15.64:0/2646684016 learned_addr learned my addr 172.21.15.64:0/2646684016 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:15:49.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.336+0000 7fbae97fa640 10 client.?.objecter ms_handle_connect 0x7fbaec0a98f0 2024-03-25T18:15:49.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.336+0000 7fbae97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:49.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.347+0000 7fbae97fa640 10 client.?.objecter ms_handle_connect 0x7fbaec1430c0 2024-03-25T18:15:49.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.347+0000 7fbae97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:49.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.349+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643070432 0 0) 0x7fbaec0abb00 con 0x7fbaec0a98f0 2024-03-25T18:15:49.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.349+0000 7fbae97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:49.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.349+0000 7fbae97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:49.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.349+0000 7fbae97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:49.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.349+0000 7fbae97fa640 10 monclient(hunting): my global_id is 4511 2024-03-25T18:15:49.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.349+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad0004eb0 con 0x7fbaec0a98f0 2024-03-25T18:15:49.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.358+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667345433 0 0) 0x7fbaec0ac910 con 0x7fbaec1430c0 2024-03-25T18:15:49.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.358+0000 7fbae97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:49.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.358+0000 7fbae97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:49.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.358+0000 7fbae97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:49.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.358+0000 7fbae97fa640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:15:49.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.358+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaec0abb00 con 0x7fbaec1430c0 2024-03-25T18:15:49.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.359+0000 7fbae97fa640 10 client.?.objecter ms_handle_connect 0x7fbaec13f710 2024-03-25T18:15:49.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.359+0000 7fbae97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:49.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.366+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 185804664 0 0) 0x7fbad0004eb0 con 0x7fbaec0a98f0 2024-03-25T18:15:49.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.366+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbaec0ac910 con 0x7fbaec0a98f0 2024-03-25T18:15:49.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.366+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbae0001a30 con 0x7fbaec0a98f0 2024-03-25T18:15:49.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.366+0000 7fbae97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:49.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.366+0000 7fbae97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:49.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.366+0000 7fbae97fa640 10 monclient(hunting): dump: 2024-03-25T18:15:49.367 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:49.367 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:49.367 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:49.367 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:49.367 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:49.367 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:49.368 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:49.368 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:49.368 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:49.368 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:49.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.366+0000 7fbae97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:49.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.377+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1025091865 0 0) 0x7fbaec0abb00 con 0x7fbaec1430c0 2024-03-25T18:15:49.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.378+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbad0004eb0 con 0x7fbaec1430c0 2024-03-25T18:15:49.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.378+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbae4001b00 con 0x7fbaec1430c0 2024-03-25T18:15:49.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.378+0000 7fbae97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:49.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.378+0000 7fbae97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:49.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.378+0000 7fbae97fa640 10 monclient(hunting): dump: 2024-03-25T18:15:49.379 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:49.379 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:49.379 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:49.379 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:49.379 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:49.379 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:49.379 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:49.379 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:49.379 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:49.379 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:49.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.379+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621183354 0 0) 0x7fbaec0b44c0 con 0x7fbaec13f710 2024-03-25T18:15:49.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.379+0000 7fbae97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:49.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.379+0000 7fbae97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:49.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.379+0000 7fbae97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:49.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.379+0000 7fbae97fa640 10 monclient(hunting): my global_id is 4506 2024-03-25T18:15:49.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.379+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaec0abb00 con 0x7fbaec13f710 2024-03-25T18:15:49.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2620650296 0 0) 0x7fbaec0ac910 con 0x7fbaec0a98f0 2024-03-25T18:15:49.385 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 >> v1:172.21.15.86:6789/0 conn(0x7fbaec1430c0 legacy=0x7fbaec1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:49.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 >> v1:172.21.15.64:6789/0 conn(0x7fbaec13f710 legacy=0x7fbaec0ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:49.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbae97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:15:49.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbae97fa640 1 monclient: found mon.c 2024-03-25T18:15:49.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbae97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:49.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbae97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:15:49.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaec0abde0 con 0x7fbaec0a98f0 2024-03-25T18:15:49.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbae97fa640 10 monclient: _finish_auth 0 2024-03-25T18:15:49.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbae97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:15:49.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbae97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:49.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbaf2bba640 5 monclient: authenticate success, global_id 4511 2024-03-25T18:15:49.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbaf2bba640 10 monclient: _renew_subs 2024-03-25T18:15:49.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbaf2bba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:15:49.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbaf2bba640 1 -- 172.21.15.64:0/2646684016 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaec138f40 con 0x7fbaec0a98f0 2024-03-25T18:15:49.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbaf2bba640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:49.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbaf2bba640 10 monclient: _renew_subs 2024-03-25T18:15:49.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbaf2bba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:15:49.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbaf2bba640 1 -- 172.21.15.64:0/2646684016 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaec1363f0 con 0x7fbaec0a98f0 2024-03-25T18:15:49.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.384+0000 7fbaf2bba640 1 librados: init done 2024-03-25T18:15:49.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.385+0000 7fbaf2bba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:49.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.385+0000 7fbaf2bba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:49.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.385+0000 7fbaf2bba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:15:49.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.385+0000 7fbaf2bba640 1 -- 172.21.15.64:0/2646684016 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaec0627b0 con 0x7fbaec0a98f0 2024-03-25T18:15:49.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.691+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbae0001c70 con 0x7fbaec0a98f0 2024-03-25T18:15:49.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.691+0000 7fbae97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.691+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbae0003a60 con 0x7fbaec0a98f0 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.691+0000 7fbae97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.691+0000 7fbae97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.691+0000 7fbae97fa640 10 monclient: dump: 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.691+0000 7fbaeaffd640 4 set_mon_vals no callback set 2024-03-25T18:15:49.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.692+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fbae0003c40 con 0x7fbaec0a98f0 2024-03-25T18:15:49.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.693+0000 7fbaebfff640 10 monclient: get_auth_request con 0x7fbad006df40 auth_method 0 2024-03-25T18:15:49.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.693+0000 7fbae97fa640 10 client.4511.objecter ms_handle_connect 0x7fbad006df40 2024-03-25T18:15:49.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.700+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fbae00548b0 con 0x7fbaec0a98f0 2024-03-25T18:15:49.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.700+0000 7fbae97fa640 10 client.4511.objecter ms_dispatch 0x7fbaec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:49.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.700+0000 7fbae97fa640 3 client.4511.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:49.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.700+0000 7fbae97fa640 3 client.4511.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:49.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.700+0000 7fbae97fa640 20 client.4511.objecter dump_active .. 0 homeless 2024-03-25T18:15:49.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.700+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbae00503c0 con 0x7fbaec0a98f0 2024-03-25T18:15:49.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.700+0000 7fbae97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:49.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.700+0000 7fbae97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:49.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.945+0000 7fbaf2bba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:49.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.945+0000 7fbaf2bba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:49.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.945+0000 7fbaf2bba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:15:49.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:49.945+0000 7fbaf2bba640 1 -- 172.21.15.64:0/2646684016 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbaec1434a0 con 0x7fbaec0a98f0 2024-03-25T18:15:50.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:50.030+0000 7fbae97fa640 1 -- 172.21.15.64:0/2646684016 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fbae00522f0 con 0x7fbaec0a98f0 2024-03-25T18:15:50.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:50.030+0000 7fbae97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:50.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:50.030+0000 7fbae97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:50.032 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:15:50.032 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:50.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:50.033+0000 7fbaf2bba640 10 librados: watch_flush enter 2024-03-25T18:15:50.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:50.033+0000 7fbaf2bba640 10 librados: watch_flush exit 2024-03-25T18:15:50.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:50.033+0000 7fbaf2bba640 20 client.4511.objecter shutdown clearing up homeless session... 2024-03-25T18:15:50.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:50.033+0000 7fbaf2bba640 10 client.4511.objecter successfully canceled tick 2024-03-25T18:15:50.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:50.033+0000 7fbaf2bba640 1 -- 172.21.15.64:0/2646684016 >> v1:172.21.15.64:6800/2915543638 conn(0x7fbad006df40 legacy=0x7fbad0070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:50.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:50.033+0000 7fbaf2bba640 10 monclient: shutdown 2024-03-25T18:15:50.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:50.033+0000 7fbaf2bba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:50.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:50.033+0000 7fbaf2bba640 1 -- 172.21.15.64:0/2646684016 >> v1:172.21.15.64:6790/0 conn(0x7fbaec0a98f0 legacy=0x7fbaec138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:50.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:50.033+0000 7fbaf2bba640 1 -- 172.21.15.64:0/2646684016 shutdown_connections 2024-03-25T18:15:50.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:50.033+0000 7fbaf2bba640 1 -- 172.21.15.64:0/2646684016 >> 172.21.15.64:0/2646684016 conn(0x7fbaec0b7ea0 msgr2=0x7fbaec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:50.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:50.033+0000 7fbaf2bba640 1 -- 172.21.15.64:0/2646684016 shutdown_connections 2024-03-25T18:15:50.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:50.033+0000 7fbaf2bba640 1 -- 172.21.15.64:0/2646684016 wait complete. 2024-03-25T18:15:50.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:50.033+0000 7fbaf2bba640 1 librados: shutdown 2024-03-25T18:15:50.044 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:56.850 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:56.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:56.962+0000 7f10c748d640 10 monclient: get_monmap_and_config 2024-03-25T18:15:56.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:56.962+0000 7f10c748d640 10 monclient: build_initial_monmap 2024-03-25T18:15:56.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:56.962+0000 7f10c748d640 10 monclient: monmap: 2024-03-25T18:15:56.965 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:15:56.965 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:56.965 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:15:56.964590+0000 2024-03-25T18:15:56.965 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:15:56.964590+0000 2024-03-25T18:15:56.965 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:56.965 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:56.965 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:15:56.965 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:15:56.965 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:15:56.966 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:56.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:56.963+0000 7f10c748d640 1 Processor -- start 2024-03-25T18:15:56.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:56.963+0000 7f10c748d640 1 -- start start 2024-03-25T18:15:56.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:56.963+0000 7f10c748d640 10 monclient: init 2024-03-25T18:15:56.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:56.964+0000 7f10c748d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:56.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:56.964+0000 7f10c748d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:56.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:56.964+0000 7f10c748d640 10 monclient(hunting): picked mon.noname-b con 0x7f10c00705b0 addr v1:172.21.15.64:6790/0 2024-03-25T18:15:56.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:56.964+0000 7f10c748d640 10 monclient(hunting): picked mon.noname-a con 0x7f10c01402d0 addr v1:172.21.15.64:6789/0 2024-03-25T18:15:56.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:56.964+0000 7f10c748d640 10 monclient(hunting): picked mon.noname-c con 0x7f10c0149ae0 addr v1:172.21.15.86:6789/0 2024-03-25T18:15:56.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:56.964+0000 7f10c748d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c00718c0 con 0x7f10c01402d0 2024-03-25T18:15:56.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:56.964+0000 7f10c748d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c00714c0 con 0x7f10c0149ae0 2024-03-25T18:15:56.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:56.964+0000 7f10c748d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c0072b50 con 0x7f10c00705b0 2024-03-25T18:15:56.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:56.964+0000 7f10c748d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:56.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:56.964+0000 7f10c748d640 10 monclient(hunting): authenticate will time out at 7343.203874s 2024-03-25T18:15:56.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:56.967+0000 7f10c5a2a640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f10c0149ae0 0x7f10c014bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:57314/0 (socket says 172.21.15.64:57314) 2024-03-25T18:15:56.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:56.967+0000 7f10c5a2a640 1 -- 172.21.15.64:0/1329702615 learned_addr learned my addr 172.21.15.64:0/1329702615 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:15:57.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.021+0000 7f10b7fff640 1 -- 172.21.15.64:0/1329702615 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364790507 0 0) 0x7f10c00714c0 con 0x7f10c0149ae0 2024-03-25T18:15:57.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.021+0000 7f10b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:57.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.021+0000 7f10b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:57.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.021+0000 7f10b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:57.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.021+0000 7f10b7fff640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:15:57.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.021+0000 7f10b7fff640 1 -- 172.21.15.64:0/1329702615 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10ac003520 con 0x7f10c0149ae0 2024-03-25T18:15:57.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.027+0000 7f10b7fff640 1 -- 172.21.15.64:0/1329702615 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356280957 0 0) 0x7f10c00718c0 con 0x7f10c01402d0 2024-03-25T18:15:57.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.027+0000 7f10b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:57.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.027+0000 7f10b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:57.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.027+0000 7f10b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:57.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.027+0000 7f10b7fff640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:15:57.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.027+0000 7f10b7fff640 1 -- 172.21.15.64:0/1329702615 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c00714c0 con 0x7f10c01402d0 2024-03-25T18:15:57.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.028+0000 7f10b7fff640 1 -- 172.21.15.64:0/1329702615 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454026026 0 0) 0x7f10c0072b50 con 0x7f10c00705b0 2024-03-25T18:15:57.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.028+0000 7f10b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:57.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.028+0000 7f10b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:57.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.028+0000 7f10b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:57.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.028+0000 7f10b7fff640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:15:57.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.028+0000 7f10b7fff640 1 -- 172.21.15.64:0/1329702615 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c00718c0 con 0x7f10c00705b0 2024-03-25T18:15:57.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.029+0000 7f10b7fff640 1 -- 172.21.15.64:0/1329702615 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3489557746 0 0) 0x7f10ac003520 con 0x7f10c0149ae0 2024-03-25T18:15:57.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.029+0000 7f10b7fff640 1 -- 172.21.15.64:0/1329702615 >> v1:172.21.15.64:6790/0 conn(0x7f10c00705b0 legacy=0x7f10c013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.029+0000 7f10b7fff640 1 -- 172.21.15.64:0/1329702615 >> v1:172.21.15.64:6789/0 conn(0x7f10c01402d0 legacy=0x7f10c0140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.029+0000 7f10b7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:57.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.029+0000 7f10b7fff640 1 monclient: found mon.noname-c 2024-03-25T18:15:57.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.029+0000 7f10b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:57.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.029+0000 7f10b7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:15:57.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.029+0000 7f10b7fff640 1 -- 172.21.15.64:0/1329702615 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c013dd10 con 0x7f10c0149ae0 2024-03-25T18:15:57.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.029+0000 7f10b7fff640 10 monclient: _finish_auth 0 2024-03-25T18:15:57.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.029+0000 7f10b7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:57.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.029+0000 7f10b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:57.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.029+0000 7f10b7fff640 1 -- 172.21.15.64:0/1329702615 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f10b8001690 con 0x7f10c0149ae0 2024-03-25T18:15:57.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.029+0000 7f10c748d640 5 monclient: authenticate success, global_id 4507 2024-03-25T18:15:57.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.029+0000 7f10c748d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:57.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.029+0000 7f10b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:57.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.029+0000 7f10b7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:57.033 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.029+0000 7f10b7fff640 10 monclient: dump: 2024-03-25T18:15:57.033 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:57.033 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:57.034 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:57.034 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:57.034 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:57.034 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:57.034 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:57.034 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:57.034 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:57.034 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:57.034 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.029+0000 7f10c748d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:57.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.049+0000 7f10b7fff640 1 -- 172.21.15.64:0/1329702615 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10b80017f0 con 0x7f10c0149ae0 2024-03-25T18:15:57.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.049+0000 7f10b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:57.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.049+0000 7f10b7fff640 1 -- 172.21.15.64:0/1329702615 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f10b8001c00 con 0x7f10c0149ae0 2024-03-25T18:15:57.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.049+0000 7f10c748d640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:57.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.049+0000 7f10c748d640 4 set_mon_vals no callback set 2024-03-25T18:15:57.051 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.049+0000 7f10b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:57.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.049+0000 7f10b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:57.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.049+0000 7f10b7fff640 10 monclient: dump: 2024-03-25T18:15:57.052 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:57.052 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:57.052 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:57.052 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:57.052 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:57.052 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:57.052 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:57.052 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:57.052 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:57.052 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:57.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.050+0000 7f10c748d640 10 monclient: shutdown 2024-03-25T18:15:57.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.050+0000 7f10c748d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:57.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.050+0000 7f10c748d640 1 -- 172.21.15.64:0/1329702615 >> v1:172.21.15.86:6789/0 conn(0x7f10c0149ae0 legacy=0x7f10c014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.050+0000 7f10c748d640 1 -- 172.21.15.64:0/1329702615 shutdown_connections 2024-03-25T18:15:57.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.050+0000 7f10c748d640 1 -- 172.21.15.64:0/1329702615 >> 172.21.15.64:0/1329702615 conn(0x7f10c00b5d10 msgr2=0x7f10c00b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:57.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.050+0000 7f10c748d640 1 -- 172.21.15.64:0/1329702615 shutdown_connections 2024-03-25T18:15:57.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.050+0000 7f10c748d640 1 -- 172.21.15.64:0/1329702615 wait complete. 2024-03-25T18:15:57.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.051+0000 7f10c748d640 10 monclient: build_initial_monmap 2024-03-25T18:15:57.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.051+0000 7f10c748d640 10 monclient: monmap: 2024-03-25T18:15:57.055 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:15:57.055 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:57.055 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:15:57.055 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:15:57.055 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:57.056 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:15:57.056 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:15:57.056 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:15:57.056 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:15:57.056 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:57.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.051+0000 7f10c748d640 1 librados: starting msgr at 2024-03-25T18:15:57.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.051+0000 7f10c748d640 1 librados: starting objecter 2024-03-25T18:15:57.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.051+0000 7f10c748d640 1 Processor -- start 2024-03-25T18:15:57.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.051+0000 7f10c748d640 1 -- start start 2024-03-25T18:15:57.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.051+0000 7f10c748d640 1 librados: setting wanted keys 2024-03-25T18:15:57.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.051+0000 7f10c748d640 1 librados: calling monclient init 2024-03-25T18:15:57.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.051+0000 7f10c748d640 10 monclient: init 2024-03-25T18:15:57.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.052+0000 7f10c748d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:57.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.052+0000 7f10c748d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:57.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.052+0000 7f10c748d640 10 monclient(hunting): picked mon.noname-b con 0x7f10c00705b0 addr v1:172.21.15.86:6789/0 2024-03-25T18:15:57.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.052+0000 7f10c748d640 10 monclient(hunting): picked mon.noname-a con 0x7f10c01402d0 addr v1:172.21.15.64:6789/0 2024-03-25T18:15:57.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.052+0000 7f10c748d640 10 monclient(hunting): picked mon.noname-c con 0x7f10c0149ae0 addr v1:172.21.15.64:6790/0 2024-03-25T18:15:57.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.052+0000 7f10c748d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c00aa3e0 con 0x7f10c01402d0 2024-03-25T18:15:57.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.052+0000 7f10c748d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c01d9ad0 con 0x7f10c00705b0 2024-03-25T18:15:57.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.052+0000 7f10c748d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c00af5f0 con 0x7f10c0149ae0 2024-03-25T18:15:57.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.052+0000 7f10c748d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:57.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.052+0000 7f10c748d640 10 monclient(hunting): authenticate will time out at 7343.292246s 2024-03-25T18:15:57.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.059+0000 7f10c5229640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f10c00705b0 0x7f10c01d91d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:57328/0 (socket says 172.21.15.64:57328) 2024-03-25T18:15:57.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.059+0000 7f10c5229640 1 -- 172.21.15.64:0/1678856954 learned_addr learned my addr 172.21.15.64:0/1678856954 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:15:57.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.060+0000 7f10b5ffb640 10 client.?.objecter ms_handle_connect 0x7f10c00705b0 2024-03-25T18:15:57.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.060+0000 7f10b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:57.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.066+0000 7f10b5ffb640 10 client.?.objecter ms_handle_connect 0x7f10c01402d0 2024-03-25T18:15:57.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.066+0000 7f10b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:57.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.069+0000 7f10b5ffb640 10 client.?.objecter ms_handle_connect 0x7f10c0149ae0 2024-03-25T18:15:57.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.069+0000 7f10b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:57.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.079+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036185514 0 0) 0x7f10c01d9ad0 con 0x7f10c00705b0 2024-03-25T18:15:57.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.079+0000 7f10b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:57.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.079+0000 7f10b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:57.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.079+0000 7f10b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:57.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.079+0000 7f10b5ffb640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:15:57.081 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.079+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a4004eb0 con 0x7f10c00705b0 2024-03-25T18:15:57.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.082+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828804357 0 0) 0x7f10c00aa3e0 con 0x7f10c01402d0 2024-03-25T18:15:57.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.082+0000 7f10b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:57.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.082+0000 7f10b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:57.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.082+0000 7f10b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:57.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.082+0000 7f10b5ffb640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:15:57.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.082+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c01d9ad0 con 0x7f10c01402d0 2024-03-25T18:15:57.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.082+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477877632 0 0) 0x7f10c00af5f0 con 0x7f10c0149ae0 2024-03-25T18:15:57.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.082+0000 7f10b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:57.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.082+0000 7f10b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:57.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.082+0000 7f10b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:57.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.082+0000 7f10b5ffb640 10 monclient(hunting): my global_id is 4517 2024-03-25T18:15:57.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.082+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c00aa3e0 con 0x7f10c0149ae0 2024-03-25T18:15:57.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.104+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1289841292 0 0) 0x7f10a4004eb0 con 0x7f10c00705b0 2024-03-25T18:15:57.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.104+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10c00af5f0 con 0x7f10c00705b0 2024-03-25T18:15:57.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.104+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f10bc001d90 con 0x7f10c00705b0 2024-03-25T18:15:57.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.104+0000 7f10b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:57.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.104+0000 7f10b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:57.107 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.104+0000 7f10b5ffb640 10 monclient(hunting): dump: 2024-03-25T18:15:57.107 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:57.107 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:57.107 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:57.107 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:57.107 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:57.107 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:57.107 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:57.107 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:57.108 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:57.108 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:57.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.104+0000 7f10b5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:57.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.104+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3127867730 0 0) 0x7f10c00aa3e0 con 0x7f10c0149ae0 2024-03-25T18:15:57.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.104+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10a4004eb0 con 0x7f10c0149ae0 2024-03-25T18:15:57.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.104+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f10b8001b00 con 0x7f10c0149ae0 2024-03-25T18:15:57.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.104+0000 7f10b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:57.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.104+0000 7f10b5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:57.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.104+0000 7f10b5ffb640 10 monclient(hunting): dump: 2024-03-25T18:15:57.108 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:57.108 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:57.108 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:57.108 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:57.109 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:57.109 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:57.109 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:57.109 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:57.109 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:57.109 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:57.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.116+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2655296366 0 0) 0x7f10c01d9ad0 con 0x7f10c01402d0 2024-03-25T18:15:57.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.116+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10c00aa3e0 con 0x7f10c01402d0 2024-03-25T18:15:57.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.116+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f10b0001970 con 0x7f10c01402d0 2024-03-25T18:15:57.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.116+0000 7f10b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:57.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.116+0000 7f10b5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:57.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.116+0000 7f10b5ffb640 10 monclient(hunting): dump: 2024-03-25T18:15:57.119 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:57.119 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:57.119 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:57.119 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:57.119 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:57.119 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:57.119 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:57.119 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:57.119 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:57.119 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:57.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1430443733 0 0) 0x7f10c00af5f0 con 0x7f10c00705b0 2024-03-25T18:15:57.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 >> v1:172.21.15.64:6790/0 conn(0x7f10c0149ae0 legacy=0x7f10c0148410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 >> v1:172.21.15.64:6789/0 conn(0x7f10c01402d0 legacy=0x7f10c00aeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10b5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:15:57.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10b5ffb640 1 monclient: found mon.b 2024-03-25T18:15:57.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:57.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10b5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:57.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c00af780 con 0x7f10c00705b0 2024-03-25T18:15:57.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10b5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:15:57.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10b5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:15:57.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:57.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10c748d640 5 monclient: authenticate success, global_id 4510 2024-03-25T18:15:57.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10c748d640 10 monclient: _renew_subs 2024-03-25T18:15:57.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10c748d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:57.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10c748d640 1 -- 172.21.15.64:0/1678856954 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c013a190 con 0x7f10c00705b0 2024-03-25T18:15:57.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10c748d640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:57.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10c748d640 10 monclient: _renew_subs 2024-03-25T18:15:57.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10c748d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:57.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10c748d640 1 -- 172.21.15.64:0/1678856954 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c013a530 con 0x7f10c00705b0 2024-03-25T18:15:57.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.137+0000 7f10c748d640 1 librados: init done 2024-03-25T18:15:57.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.138+0000 7f10c748d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:57.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.138+0000 7f10c748d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:57.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.138+0000 7f10c748d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:57.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.138+0000 7f10c748d640 1 -- 172.21.15.64:0/1678856954 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c01d9c60 con 0x7f10c00705b0 2024-03-25T18:15:57.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.425+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10bc001ff0 con 0x7f10c00705b0 2024-03-25T18:15:57.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.425+0000 7f10b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:57.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.425+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f10bc003de0 con 0x7f10c00705b0 2024-03-25T18:15:57.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.425+0000 7f10b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:57.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.425+0000 7f10b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:57.427 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.425+0000 7f10b5ffb640 10 monclient: dump: 2024-03-25T18:15:57.428 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:15:57.428 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:15:57.428 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:57.428 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:15:57.428 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:57.428 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:15:57.428 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:15:57.428 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:15:57.428 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:15:57.428 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:15:57.428 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.425+0000 7f10b77fe640 4 set_mon_vals no callback set 2024-03-25T18:15:57.429 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.426+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f10bc004000 con 0x7f10c00705b0 2024-03-25T18:15:57.429 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.426+0000 7f10c4a28640 10 monclient: get_auth_request con 0x7f10a406e0f0 auth_method 0 2024-03-25T18:15:57.429 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.427+0000 7f10b5ffb640 10 client.4510.objecter ms_handle_connect 0x7f10a406e0f0 2024-03-25T18:15:57.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.430+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f10bc054be0 con 0x7f10c00705b0 2024-03-25T18:15:57.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.430+0000 7f10b5ffb640 10 client.4510.objecter ms_dispatch 0x7f10c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:57.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.430+0000 7f10b5ffb640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:57.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.430+0000 7f10b5ffb640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:57.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.431+0000 7f10b5ffb640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-25T18:15:57.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.431+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f10bc0506f0 con 0x7f10c00705b0 2024-03-25T18:15:57.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.431+0000 7f10b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:57.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.431+0000 7f10b5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:57.549 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:06:40.661 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:57.601 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:06:40.714 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:57.629 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:06:40.741 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:57.670 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:06:40.749 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:57.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.671+0000 7f10c748d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:57.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.671+0000 7f10c748d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:57.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.671+0000 7f10c748d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:15:57.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.671+0000 7f10c748d640 1 -- 172.21.15.64:0/1678856954 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10c013a320 con 0x7f10c00705b0 2024-03-25T18:15:57.679 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:06:40.757 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:57.683 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:06:40.796 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:57.693 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:06:40.806 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:57.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.715+0000 7f10b5ffb640 1 -- 172.21.15.64:0/1678856954 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f10bc052620 con 0x7f10c00705b0 2024-03-25T18:15:57.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.715+0000 7f10b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:57.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.715+0000 7f10b5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:57.717 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:15:57.718 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:57.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.717+0000 7f10c748d640 10 librados: watch_flush enter 2024-03-25T18:15:57.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.717+0000 7f10c748d640 10 librados: watch_flush exit 2024-03-25T18:15:57.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.717+0000 7f10c748d640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-25T18:15:57.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.717+0000 7f10c748d640 10 client.4510.objecter successfully canceled tick 2024-03-25T18:15:57.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.717+0000 7f10c748d640 1 -- 172.21.15.64:0/1678856954 >> v1:172.21.15.64:6800/2915543638 conn(0x7f10a406e0f0 legacy=0x7f10a40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.717+0000 7f10c748d640 10 monclient: shutdown 2024-03-25T18:15:57.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.717+0000 7f10c748d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:57.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.717+0000 7f10c748d640 1 -- 172.21.15.64:0/1678856954 >> v1:172.21.15.86:6789/0 conn(0x7f10c00705b0 legacy=0x7f10c01d91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.717+0000 7f10c748d640 1 -- 172.21.15.64:0/1678856954 shutdown_connections 2024-03-25T18:15:57.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.717+0000 7f10c748d640 1 -- 172.21.15.64:0/1678856954 >> 172.21.15.64:0/1678856954 conn(0x7f10c00b5d10 msgr2=0x7f10c00b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:57.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.717+0000 7f10c748d640 1 -- 172.21.15.64:0/1678856954 shutdown_connections 2024-03-25T18:15:57.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.718+0000 7f10c748d640 1 -- 172.21.15.64:0/1678856954 wait complete. 2024-03-25T18:15:57.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:15:57.718+0000 7f10c748d640 1 librados: shutdown 2024-03-25T18:15:57.731 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:57.734 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:06:40.813 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:57.741 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:06:40.853 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:04.539 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:04.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.654+0000 7f93df3e8640 10 monclient: get_monmap_and_config 2024-03-25T18:16:04.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.654+0000 7f93df3e8640 10 monclient: build_initial_monmap 2024-03-25T18:16:04.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.654+0000 7f93df3e8640 10 monclient: monmap: 2024-03-25T18:16:04.655 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:16:04.655 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:04.655 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:16:04.655042+0000 2024-03-25T18:16:04.655 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:16:04.655042+0000 2024-03-25T18:16:04.655 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:04.655 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:04.656 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:16:04.656 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:16:04.656 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:16:04.656 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:04.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.654+0000 7f93df3e8640 1 Processor -- start 2024-03-25T18:16:04.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.655+0000 7f93df3e8640 1 -- start start 2024-03-25T18:16:04.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.655+0000 7f93df3e8640 10 monclient: init 2024-03-25T18:16:04.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.655+0000 7f93df3e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:04.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.655+0000 7f93df3e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:04.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.655+0000 7f93df3e8640 10 monclient(hunting): picked mon.noname-c con 0x7f93d80a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:16:04.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.655+0000 7f93df3e8640 10 monclient(hunting): picked mon.noname-a con 0x7f93d813f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:16:04.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.655+0000 7f93df3e8640 10 monclient(hunting): picked mon.noname-b con 0x7f93d81430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:16:04.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.655+0000 7f93df3e8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d8142520 con 0x7f93d813f710 2024-03-25T18:16:04.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.655+0000 7f93df3e8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d80b28a0 con 0x7f93d80a98f0 2024-03-25T18:16:04.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.655+0000 7f93df3e8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d80b2a30 con 0x7f93d81430c0 2024-03-25T18:16:04.657 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.655+0000 7f93df3e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:04.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.655+0000 7f93df3e8640 10 monclient(hunting): authenticate will time out at 7350.894206s 2024-03-25T18:16:04.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.663+0000 7f93dd985640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f93d81430c0 0x7f93d814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:49844/0 (socket says 172.21.15.64:49844) 2024-03-25T18:16:04.665 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.663+0000 7f93dd985640 1 -- 172.21.15.64:0/2623187632 learned_addr learned my addr 172.21.15.64:0/2623187632 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:16:04.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.675+0000 7f93cffff640 1 -- 172.21.15.64:0/2623187632 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983959783 0 0) 0x7f93d8142520 con 0x7f93d813f710 2024-03-25T18:16:04.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.675+0000 7f93cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:04.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.675+0000 7f93cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:04.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.675+0000 7f93cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:04.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.675+0000 7f93cffff640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:16:04.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.675+0000 7f93cffff640 1 -- 172.21.15.64:0/2623187632 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93c4003520 con 0x7f93d813f710 2024-03-25T18:16:04.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.684+0000 7f93cffff640 1 -- 172.21.15.64:0/2623187632 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974938556 0 0) 0x7f93d80b2a30 con 0x7f93d81430c0 2024-03-25T18:16:04.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.684+0000 7f93cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:04.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.684+0000 7f93cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:04.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.684+0000 7f93cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:04.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.684+0000 7f93cffff640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:16:04.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.684+0000 7f93cffff640 1 -- 172.21.15.64:0/2623187632 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93d8142520 con 0x7f93d81430c0 2024-03-25T18:16:04.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.684+0000 7f93cffff640 1 -- 172.21.15.64:0/2623187632 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862738463 0 0) 0x7f93d80b28a0 con 0x7f93d80a98f0 2024-03-25T18:16:04.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.684+0000 7f93cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:04.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.684+0000 7f93cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:04.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.684+0000 7f93cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:04.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.684+0000 7f93cffff640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:16:04.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.684+0000 7f93cffff640 1 -- 172.21.15.64:0/2623187632 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93d80b2a30 con 0x7f93d80a98f0 2024-03-25T18:16:04.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.689+0000 7f93cffff640 1 -- 172.21.15.64:0/2623187632 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3963468162 0 0) 0x7f93d80b2a30 con 0x7f93d80a98f0 2024-03-25T18:16:04.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.689+0000 7f93cffff640 1 -- 172.21.15.64:0/2623187632 >> v1:172.21.15.64:6790/0 conn(0x7f93d81430c0 legacy=0x7f93d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.689+0000 7f93cffff640 1 -- 172.21.15.64:0/2623187632 >> v1:172.21.15.64:6789/0 conn(0x7f93d813f710 legacy=0x7f93d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.689+0000 7f93cffff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:04.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.689+0000 7f93cffff640 1 monclient: found mon.noname-c 2024-03-25T18:16:04.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.689+0000 7f93cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:04.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.689+0000 7f93cffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:16:04.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.689+0000 7f93cffff640 1 -- 172.21.15.64:0/2623187632 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d80b44c0 con 0x7f93d80a98f0 2024-03-25T18:16:04.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.689+0000 7f93cffff640 10 monclient: _finish_auth 0 2024-03-25T18:16:04.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.689+0000 7f93cffff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:04.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.689+0000 7f93cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:04.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.689+0000 7f93df3e8640 5 monclient: authenticate success, global_id 4513 2024-03-25T18:16:04.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.689+0000 7f93df3e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:04.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.692+0000 7f93cffff640 1 -- 172.21.15.64:0/2623187632 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f93d40014f0 con 0x7f93d80a98f0 2024-03-25T18:16:04.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.692+0000 7f93cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:04.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.692+0000 7f93cffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:04.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.692+0000 7f93cffff640 10 monclient: dump: 2024-03-25T18:16:04.693 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:04.693 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:04.693 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:04.693 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:04.693 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:04.693 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:04.693 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:04.693 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:04.693 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:04.694 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:04.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.692+0000 7f93df3e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:04.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.696+0000 7f93cffff640 1 -- 172.21.15.64:0/2623187632 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93d40017c0 con 0x7f93d80a98f0 2024-03-25T18:16:04.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.696+0000 7f93cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:04.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.696+0000 7f93cffff640 1 -- 172.21.15.64:0/2623187632 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f93d4001b50 con 0x7f93d80a98f0 2024-03-25T18:16:04.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.696+0000 7f93cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:04.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.696+0000 7f93cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:04.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.696+0000 7f93cffff640 10 monclient: dump: 2024-03-25T18:16:04.697 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:04.698 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:04.698 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:04.698 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:04.698 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:04.698 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:04.698 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:04.698 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:04.698 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:04.698 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:04.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.696+0000 7f93df3e8640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:04.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.696+0000 7f93df3e8640 4 set_mon_vals no callback set 2024-03-25T18:16:04.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.697+0000 7f93df3e8640 10 monclient: shutdown 2024-03-25T18:16:04.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.697+0000 7f93df3e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:04.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.697+0000 7f93df3e8640 1 -- 172.21.15.64:0/2623187632 >> v1:172.21.15.86:6789/0 conn(0x7f93d80a98f0 legacy=0x7f93d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.697+0000 7f93df3e8640 1 -- 172.21.15.64:0/2623187632 shutdown_connections 2024-03-25T18:16:04.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.697+0000 7f93df3e8640 1 -- 172.21.15.64:0/2623187632 >> 172.21.15.64:0/2623187632 conn(0x7f93d80b7ea0 msgr2=0x7f93d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:04.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.697+0000 7f93df3e8640 1 -- 172.21.15.64:0/2623187632 shutdown_connections 2024-03-25T18:16:04.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.697+0000 7f93df3e8640 1 -- 172.21.15.64:0/2623187632 wait complete. 2024-03-25T18:16:04.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.698+0000 7f93df3e8640 10 monclient: build_initial_monmap 2024-03-25T18:16:04.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.698+0000 7f93df3e8640 10 monclient: monmap: 2024-03-25T18:16:04.701 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:16:04.701 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:04.701 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:16:04.701 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:16:04.701 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:04.701 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:16:04.701 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:16:04.702 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:16:04.702 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:16:04.702 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:04.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.698+0000 7f93df3e8640 1 librados: starting msgr at 2024-03-25T18:16:04.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.698+0000 7f93df3e8640 1 librados: starting objecter 2024-03-25T18:16:04.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.698+0000 7f93df3e8640 1 Processor -- start 2024-03-25T18:16:04.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.698+0000 7f93df3e8640 1 -- start start 2024-03-25T18:16:04.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.698+0000 7f93df3e8640 1 librados: setting wanted keys 2024-03-25T18:16:04.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.698+0000 7f93df3e8640 1 librados: calling monclient init 2024-03-25T18:16:04.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.698+0000 7f93df3e8640 10 monclient: init 2024-03-25T18:16:04.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.699+0000 7f93df3e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:04.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.699+0000 7f93df3e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:04.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.699+0000 7f93df3e8640 10 monclient(hunting): picked mon.noname-b con 0x7f93d80a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:16:04.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.699+0000 7f93df3e8640 10 monclient(hunting): picked mon.noname-c con 0x7f93d813f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:16:04.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.699+0000 7f93df3e8640 10 monclient(hunting): picked mon.noname-a con 0x7f93d81430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:16:04.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.699+0000 7f93df3e8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d806fa80 con 0x7f93d81430c0 2024-03-25T18:16:04.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.699+0000 7f93df3e8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d806fcf0 con 0x7f93d80a98f0 2024-03-25T18:16:04.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.699+0000 7f93df3e8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93d814e850 con 0x7f93d813f710 2024-03-25T18:16:04.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.699+0000 7f93df3e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:04.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.699+0000 7f93df3e8640 10 monclient(hunting): authenticate will time out at 7350.938099s 2024-03-25T18:16:04.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.712+0000 7f93dd985640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f93d81430c0 0x7f93d81df530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:40092/0 (socket says 172.21.15.64:40092) 2024-03-25T18:16:04.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.712+0000 7f93dd985640 1 -- 172.21.15.64:0/3315130022 learned_addr learned my addr 172.21.15.64:0/3315130022 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:16:04.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.714+0000 7f93cdffb640 10 client.?.objecter ms_handle_connect 0x7f93d81430c0 2024-03-25T18:16:04.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.714+0000 7f93cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:04.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.720+0000 7f93cdffb640 10 client.?.objecter ms_handle_connect 0x7f93d813f710 2024-03-25T18:16:04.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.720+0000 7f93cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:04.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.721+0000 7f93cdffb640 10 client.?.objecter ms_handle_connect 0x7f93d80a98f0 2024-03-25T18:16:04.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.721+0000 7f93cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:04.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.730+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156335788 0 0) 0x7f93d806fa80 con 0x7f93d81430c0 2024-03-25T18:16:04.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.730+0000 7f93cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:04.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.730+0000 7f93cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:04.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.730+0000 7f93cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:04.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.730+0000 7f93cdffb640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:16:04.731 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.730+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93bc004eb0 con 0x7f93d81430c0 2024-03-25T18:16:04.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.734+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626253997 0 0) 0x7f93d814e850 con 0x7f93d813f710 2024-03-25T18:16:04.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.734+0000 7f93cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:04.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.734+0000 7f93cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:04.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.734+0000 7f93cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:04.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.734+0000 7f93cdffb640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:16:04.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.734+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93d806fa80 con 0x7f93d813f710 2024-03-25T18:16:04.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.735+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664088783 0 0) 0x7f93d806fcf0 con 0x7f93d80a98f0 2024-03-25T18:16:04.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.735+0000 7f93cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:04.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.735+0000 7f93cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:04.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.735+0000 7f93cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:04.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.735+0000 7f93cdffb640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:16:04.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.735+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93d814e850 con 0x7f93d80a98f0 2024-03-25T18:16:04.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.752+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3930884768 0 0) 0x7f93bc004eb0 con 0x7f93d81430c0 2024-03-25T18:16:04.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.752+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93d806fcf0 con 0x7f93d81430c0 2024-03-25T18:16:04.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.752+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f93d0001b70 con 0x7f93d81430c0 2024-03-25T18:16:04.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.752+0000 7f93cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:04.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.752+0000 7f93cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:04.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.752+0000 7f93cdffb640 10 monclient(hunting): dump: 2024-03-25T18:16:04.753 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:04.753 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:04.753 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:04.753 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:04.754 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:04.754 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:04.754 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:04.754 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:04.754 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:04.754 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:04.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.752+0000 7f93cdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:04.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.753+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1726290916 0 0) 0x7f93d814e850 con 0x7f93d80a98f0 2024-03-25T18:16:04.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.753+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93bc004eb0 con 0x7f93d80a98f0 2024-03-25T18:16:04.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.753+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f93d40014f0 con 0x7f93d80a98f0 2024-03-25T18:16:04.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.753+0000 7f93cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:04.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.753+0000 7f93cdffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:04.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.753+0000 7f93cdffb640 10 monclient(hunting): dump: 2024-03-25T18:16:04.755 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:04.755 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:04.755 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:04.755 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:04.755 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:04.755 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:04.755 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:04.756 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:04.756 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:04.756 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:04.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.761+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 374917164 0 0) 0x7f93d806fa80 con 0x7f93d813f710 2024-03-25T18:16:04.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.761+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93d814e850 con 0x7f93d813f710 2024-03-25T18:16:04.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.761+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f93c8001910 con 0x7f93d813f710 2024-03-25T18:16:04.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.761+0000 7f93cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:04.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.761+0000 7f93cdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:04.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.761+0000 7f93cdffb640 10 monclient(hunting): dump: 2024-03-25T18:16:04.762 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:04.762 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:04.762 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:04.762 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:04.763 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:04.763 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:04.763 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:04.763 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:04.763 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:04.763 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:04.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1301480426 0 0) 0x7f93d806fcf0 con 0x7f93d81430c0 2024-03-25T18:16:04.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 >> v1:172.21.15.64:6790/0 conn(0x7f93d813f710 legacy=0x7f93d81dee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 >> v1:172.21.15.86:6789/0 conn(0x7f93d80a98f0 legacy=0x7f93d8071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93cdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:04.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93cdffb640 1 monclient: found mon.a 2024-03-25T18:16:04.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:04.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93cdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:04.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d814ed70 con 0x7f93d81430c0 2024-03-25T18:16:04.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93cdffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:04.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93cdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:04.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:04.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93df3e8640 5 monclient: authenticate success, global_id 4518 2024-03-25T18:16:04.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93df3e8640 10 monclient: _renew_subs 2024-03-25T18:16:04.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93df3e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:04.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93df3e8640 1 -- 172.21.15.64:0/3315130022 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d814bec0 con 0x7f93d81430c0 2024-03-25T18:16:04.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93df3e8640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:04.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93df3e8640 10 monclient: _renew_subs 2024-03-25T18:16:04.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93df3e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:04.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93df3e8640 1 -- 172.21.15.64:0/3315130022 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d806e080 con 0x7f93d81430c0 2024-03-25T18:16:04.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.765+0000 7f93df3e8640 1 librados: init done 2024-03-25T18:16:04.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.766+0000 7f93df3e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:04.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.766+0000 7f93df3e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:04.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.766+0000 7f93df3e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:04.768 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:04.766+0000 7f93df3e8640 1 -- 172.21.15.64:0/3315130022 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d8074cc0 con 0x7f93d81430c0 2024-03-25T18:16:05.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.076+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93d0001fe0 con 0x7f93d81430c0 2024-03-25T18:16:05.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.076+0000 7f93cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:05.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.076+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f93d0004d00 con 0x7f93d81430c0 2024-03-25T18:16:05.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.076+0000 7f93cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:05.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.076+0000 7f93cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:05.078 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.076+0000 7f93cdffb640 10 monclient: dump: 2024-03-25T18:16:05.078 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:05.078 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:05.078 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:05.078 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:05.078 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:05.078 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:05.078 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:05.078 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:05.078 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:05.078 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:05.078 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.076+0000 7f93cf7fe640 4 set_mon_vals no callback set 2024-03-25T18:16:05.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.077+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f93d001ad10 con 0x7f93d81430c0 2024-03-25T18:16:05.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.078+0000 7f93dd184640 10 monclient: get_auth_request con 0x7f93bc06e0b0 auth_method 0 2024-03-25T18:16:05.079 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.078+0000 7f93cdffb640 10 client.4518.objecter ms_handle_connect 0x7f93bc06e0b0 2024-03-25T18:16:05.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.084+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f93d0054ae0 con 0x7f93d81430c0 2024-03-25T18:16:05.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.084+0000 7f93cdffb640 10 client.4518.objecter ms_dispatch 0x7f93d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:05.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.084+0000 7f93cdffb640 3 client.4518.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:05.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.084+0000 7f93cdffb640 3 client.4518.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:05.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.084+0000 7f93cdffb640 20 client.4518.objecter dump_active .. 0 homeless 2024-03-25T18:16:05.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.084+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f93d0050420 con 0x7f93d81430c0 2024-03-25T18:16:05.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.084+0000 7f93cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:05.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.084+0000 7f93cdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:05.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.337+0000 7f93df3e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:05.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.337+0000 7f93df3e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:05.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.337+0000 7f93df3e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:05.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.337+0000 7f93df3e8640 1 -- 172.21.15.64:0/3315130022 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93d80627b0 con 0x7f93d81430c0 2024-03-25T18:16:05.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.361+0000 7f93cdffb640 1 -- 172.21.15.64:0/3315130022 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f93d0052520 con 0x7f93d81430c0 2024-03-25T18:16:05.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.361+0000 7f93cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:05.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.361+0000 7f93cdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:05.362 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:16:05.362 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:05.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.363+0000 7f93df3e8640 10 librados: watch_flush enter 2024-03-25T18:16:05.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.363+0000 7f93df3e8640 10 librados: watch_flush exit 2024-03-25T18:16:05.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.363+0000 7f93df3e8640 20 client.4518.objecter shutdown clearing up homeless session... 2024-03-25T18:16:05.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.363+0000 7f93df3e8640 10 client.4518.objecter successfully canceled tick 2024-03-25T18:16:05.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.363+0000 7f93df3e8640 1 -- 172.21.15.64:0/3315130022 >> v1:172.21.15.64:6800/2915543638 conn(0x7f93bc06e0b0 legacy=0x7f93bc070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:05.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.363+0000 7f93df3e8640 10 monclient: shutdown 2024-03-25T18:16:05.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.363+0000 7f93df3e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:05.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.363+0000 7f93df3e8640 1 -- 172.21.15.64:0/3315130022 >> v1:172.21.15.64:6789/0 conn(0x7f93d81430c0 legacy=0x7f93d81df530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:05.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.364+0000 7f93df3e8640 1 -- 172.21.15.64:0/3315130022 shutdown_connections 2024-03-25T18:16:05.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.364+0000 7f93df3e8640 1 -- 172.21.15.64:0/3315130022 >> 172.21.15.64:0/3315130022 conn(0x7f93d80b7ea0 msgr2=0x7f93d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:05.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.364+0000 7f93df3e8640 1 -- 172.21.15.64:0/3315130022 shutdown_connections 2024-03-25T18:16:05.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.364+0000 7f93df3e8640 1 -- 172.21.15.64:0/3315130022 wait complete. 2024-03-25T18:16:05.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:05.364+0000 7f93df3e8640 1 librados: shutdown 2024-03-25T18:16:05.377 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:12.182 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:12.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.295+0000 7f2e620d3640 10 monclient: get_monmap_and_config 2024-03-25T18:16:12.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.295+0000 7f2e620d3640 10 monclient: build_initial_monmap 2024-03-25T18:16:12.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.295+0000 7f2e620d3640 10 monclient: monmap: 2024-03-25T18:16:12.296 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:16:12.297 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:12.297 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:16:12.296700+0000 2024-03-25T18:16:12.297 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:16:12.296700+0000 2024-03-25T18:16:12.297 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:12.297 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:12.297 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:16:12.297 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:16:12.297 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:16:12.297 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:12.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.296+0000 7f2e620d3640 1 Processor -- start 2024-03-25T18:16:12.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.296+0000 7f2e620d3640 1 -- start start 2024-03-25T18:16:12.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.296+0000 7f2e620d3640 10 monclient: init 2024-03-25T18:16:12.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.297+0000 7f2e620d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:12.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.297+0000 7f2e620d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:12.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.297+0000 7f2e620d3640 10 monclient(hunting): picked mon.noname-c con 0x7f2e5c0a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:16:12.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.297+0000 7f2e620d3640 10 monclient(hunting): picked mon.noname-a con 0x7f2e5c13f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:16:12.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.297+0000 7f2e620d3640 10 monclient(hunting): picked mon.noname-b con 0x7f2e5c1430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:16:12.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.297+0000 7f2e620d3640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c142520 con 0x7f2e5c13f710 2024-03-25T18:16:12.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.297+0000 7f2e620d3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c0b28a0 con 0x7f2e5c0a98f0 2024-03-25T18:16:12.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.297+0000 7f2e620d3640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c0b2a30 con 0x7f2e5c1430c0 2024-03-25T18:16:12.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.297+0000 7f2e620d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:12.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.297+0000 7f2e620d3640 10 monclient(hunting): authenticate will time out at 7358.536344s 2024-03-25T18:16:12.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.309+0000 7f2e5b7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2e5c0a98f0 0x7f2e5c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:59612/0 (socket says 172.21.15.64:59612) 2024-03-25T18:16:12.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.309+0000 7f2e5b7fe640 1 -- 172.21.15.64:0/212257686 learned_addr learned my addr 172.21.15.64:0/212257686 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:16:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.326+0000 7f2e5a7fc640 1 -- 172.21.15.64:0/212257686 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924912497 0 0) 0x7f2e5c0b2a30 con 0x7f2e5c1430c0 2024-03-25T18:16:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.326+0000 7f2e5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.326+0000 7f2e5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.326+0000 7f2e5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.326+0000 7f2e5a7fc640 10 monclient(hunting): my global_id is 4526 2024-03-25T18:16:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.326+0000 7f2e5a7fc640 1 -- 172.21.15.64:0/212257686 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e44003520 con 0x7f2e5c1430c0 2024-03-25T18:16:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.326+0000 7f2e5a7fc640 1 -- 172.21.15.64:0/212257686 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422345493 0 0) 0x7f2e5c0b28a0 con 0x7f2e5c0a98f0 2024-03-25T18:16:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.326+0000 7f2e5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.326+0000 7f2e5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.326+0000 7f2e5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.326+0000 7f2e5a7fc640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:16:12.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.326+0000 7f2e5a7fc640 1 -- 172.21.15.64:0/212257686 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e5c0b2a30 con 0x7f2e5c0a98f0 2024-03-25T18:16:12.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.328+0000 7f2e5a7fc640 1 -- 172.21.15.64:0/212257686 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484492098 0 0) 0x7f2e5c142520 con 0x7f2e5c13f710 2024-03-25T18:16:12.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.328+0000 7f2e5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:12.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.328+0000 7f2e5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:12.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.328+0000 7f2e5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:12.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.328+0000 7f2e5a7fc640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:16:12.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.328+0000 7f2e5a7fc640 1 -- 172.21.15.64:0/212257686 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e5c0b28a0 con 0x7f2e5c13f710 2024-03-25T18:16:12.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.339+0000 7f2e5a7fc640 1 -- 172.21.15.64:0/212257686 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 858993943 0 0) 0x7f2e5c0b2a30 con 0x7f2e5c0a98f0 2024-03-25T18:16:12.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.339+0000 7f2e5a7fc640 1 -- 172.21.15.64:0/212257686 >> v1:172.21.15.64:6790/0 conn(0x7f2e5c1430c0 legacy=0x7f2e5c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:12.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.339+0000 7f2e5a7fc640 1 -- 172.21.15.64:0/212257686 >> v1:172.21.15.64:6789/0 conn(0x7f2e5c13f710 legacy=0x7f2e5c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:12.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.339+0000 7f2e5a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:12.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.339+0000 7f2e5a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:16:12.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.339+0000 7f2e5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:12.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.339+0000 7f2e5a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:16:12.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.339+0000 7f2e5a7fc640 1 -- 172.21.15.64:0/212257686 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5c0b44c0 con 0x7f2e5c0a98f0 2024-03-25T18:16:12.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.339+0000 7f2e5a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:12.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.339+0000 7f2e5a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:12.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.339+0000 7f2e5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:12.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.339+0000 7f2e5a7fc640 1 -- 172.21.15.64:0/212257686 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2e4c001670 con 0x7f2e5c0a98f0 2024-03-25T18:16:12.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.339+0000 7f2e5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:12.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.339+0000 7f2e5a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:12.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.339+0000 7f2e5a7fc640 10 monclient: dump: 2024-03-25T18:16:12.341 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:12.341 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:12.341 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:12.341 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:12.341 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:12.341 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:12.342 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:12.342 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:12.342 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:12.342 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:12.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.339+0000 7f2e620d3640 5 monclient: authenticate success, global_id 4519 2024-03-25T18:16:12.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.339+0000 7f2e620d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:12.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.347+0000 7f2e5a7fc640 1 -- 172.21.15.64:0/212257686 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e4c0017d0 con 0x7f2e5c0a98f0 2024-03-25T18:16:12.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.347+0000 7f2e5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:12.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.347+0000 7f2e5a7fc640 1 -- 172.21.15.64:0/212257686 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2e4c001be0 con 0x7f2e5c0a98f0 2024-03-25T18:16:12.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.347+0000 7f2e5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:12.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.347+0000 7f2e5a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:12.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.347+0000 7f2e5a7fc640 10 monclient: dump: 2024-03-25T18:16:12.348 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:12.348 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:12.348 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:12.348 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:12.348 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:12.348 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:12.349 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:12.349 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:12.349 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:12.349 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:12.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.347+0000 7f2e620d3640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:12.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.347+0000 7f2e620d3640 4 set_mon_vals no callback set 2024-03-25T18:16:12.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.348+0000 7f2e620d3640 10 monclient: shutdown 2024-03-25T18:16:12.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.348+0000 7f2e620d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:12.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.348+0000 7f2e620d3640 1 -- 172.21.15.64:0/212257686 >> v1:172.21.15.86:6789/0 conn(0x7f2e5c0a98f0 legacy=0x7f2e5c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:12.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.348+0000 7f2e620d3640 1 -- 172.21.15.64:0/212257686 shutdown_connections 2024-03-25T18:16:12.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.348+0000 7f2e620d3640 1 -- 172.21.15.64:0/212257686 >> 172.21.15.64:0/212257686 conn(0x7f2e5c0b7ea0 msgr2=0x7f2e5c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:12.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.348+0000 7f2e620d3640 1 -- 172.21.15.64:0/212257686 shutdown_connections 2024-03-25T18:16:12.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.348+0000 7f2e620d3640 1 -- 172.21.15.64:0/212257686 wait complete. 2024-03-25T18:16:12.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.349+0000 7f2e620d3640 10 monclient: build_initial_monmap 2024-03-25T18:16:12.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.349+0000 7f2e620d3640 10 monclient: monmap: 2024-03-25T18:16:12.351 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:16:12.351 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:12.351 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:16:12.351 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:16:12.351 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:12.351 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:16:12.351 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:16:12.351 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:16:12.351 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:16:12.351 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:12.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.349+0000 7f2e620d3640 1 librados: starting msgr at 2024-03-25T18:16:12.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.349+0000 7f2e620d3640 1 librados: starting objecter 2024-03-25T18:16:12.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.349+0000 7f2e620d3640 1 Processor -- start 2024-03-25T18:16:12.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.349+0000 7f2e620d3640 1 -- start start 2024-03-25T18:16:12.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.349+0000 7f2e620d3640 1 librados: setting wanted keys 2024-03-25T18:16:12.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.349+0000 7f2e620d3640 1 librados: calling monclient init 2024-03-25T18:16:12.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.349+0000 7f2e620d3640 10 monclient: init 2024-03-25T18:16:12.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.350+0000 7f2e620d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:12.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.350+0000 7f2e620d3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:12.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.350+0000 7f2e620d3640 10 monclient(hunting): picked mon.noname-c con 0x7f2e5c0a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:16:12.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.350+0000 7f2e620d3640 10 monclient(hunting): picked mon.noname-b con 0x7f2e5c13f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:16:12.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.350+0000 7f2e620d3640 10 monclient(hunting): picked mon.noname-a con 0x7f2e5c1430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:16:12.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.350+0000 7f2e620d3640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c137f50 con 0x7f2e5c1430c0 2024-03-25T18:16:12.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.350+0000 7f2e620d3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c139830 con 0x7f2e5c13f710 2024-03-25T18:16:12.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.350+0000 7f2e620d3640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c0aba50 con 0x7f2e5c0a98f0 2024-03-25T18:16:12.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.350+0000 7f2e620d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:12.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.350+0000 7f2e620d3640 10 monclient(hunting): authenticate will time out at 7358.589140s 2024-03-25T18:16:12.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.360+0000 7f2e5affd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2e5c13f710 0x7f2e5c135b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:59650/0 (socket says 172.21.15.64:59650) 2024-03-25T18:16:12.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.360+0000 7f2e5affd640 1 -- 172.21.15.64:0/3521717888 learned_addr learned my addr 172.21.15.64:0/3521717888 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:16:12.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.365+0000 7f2e43fff640 10 client.?.objecter ms_handle_connect 0x7f2e5c1430c0 2024-03-25T18:16:12.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.365+0000 7f2e43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:12.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.366+0000 7f2e43fff640 10 client.?.objecter ms_handle_connect 0x7f2e5c13f710 2024-03-25T18:16:12.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.366+0000 7f2e43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:12.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.371+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179692796 0 0) 0x7f2e5c137f50 con 0x7f2e5c1430c0 2024-03-25T18:16:12.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.371+0000 7f2e43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:12.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.371+0000 7f2e43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:12.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.371+0000 7f2e43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:12.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.371+0000 7f2e43fff640 10 monclient(hunting): my global_id is 4524 2024-03-25T18:16:12.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.371+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e48004eb0 con 0x7f2e5c1430c0 2024-03-25T18:16:12.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.379+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805829231 0 0) 0x7f2e5c139830 con 0x7f2e5c13f710 2024-03-25T18:16:12.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.379+0000 7f2e43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:12.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.379+0000 7f2e43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:12.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.379+0000 7f2e43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:12.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.379+0000 7f2e43fff640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:16:12.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.379+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e5c137f50 con 0x7f2e5c13f710 2024-03-25T18:16:12.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.381+0000 7f2e43fff640 10 client.?.objecter ms_handle_connect 0x7f2e5c0a98f0 2024-03-25T18:16:12.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.381+0000 7f2e43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:12.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.385+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1757434348 0 0) 0x7f2e48004eb0 con 0x7f2e5c1430c0 2024-03-25T18:16:12.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.385+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e5c139830 con 0x7f2e5c1430c0 2024-03-25T18:16:12.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.385+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2e54001af0 con 0x7f2e5c1430c0 2024-03-25T18:16:12.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.385+0000 7f2e43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:12.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.385+0000 7f2e43fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:12.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.385+0000 7f2e43fff640 10 monclient(hunting): dump: 2024-03-25T18:16:12.387 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:12.387 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:12.387 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:12.387 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:12.387 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:12.387 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:12.387 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:12.387 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:12.387 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:12.387 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:12.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.385+0000 7f2e43fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:12.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.393+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153482815 0 0) 0x7f2e5c0aba50 con 0x7f2e5c0a98f0 2024-03-25T18:16:12.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.393+0000 7f2e43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:12.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.393+0000 7f2e43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:12.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.393+0000 7f2e43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:12.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.393+0000 7f2e43fff640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:16:12.394 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.393+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e48004eb0 con 0x7f2e5c0a98f0 2024-03-25T18:16:12.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1847178607 0 0) 0x7f2e5c139830 con 0x7f2e5c1430c0 2024-03-25T18:16:12.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 >> v1:172.21.15.64:6790/0 conn(0x7f2e5c0a98f0 legacy=0x7f2e5c134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:12.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 >> v1:172.21.15.86:6789/0 conn(0x7f2e5c13f710 legacy=0x7f2e5c135b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:12.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e43fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:12.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e43fff640 1 monclient: found mon.a 2024-03-25T18:16:12.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:12.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e43fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:12.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5c0abd30 con 0x7f2e5c1430c0 2024-03-25T18:16:12.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e43fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:12.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e43fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:12.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:12.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e620d3640 5 monclient: authenticate success, global_id 4524 2024-03-25T18:16:12.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e620d3640 10 monclient: _renew_subs 2024-03-25T18:16:12.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e620d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:12.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e620d3640 1 -- 172.21.15.64:0/3521717888 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e5c137a00 con 0x7f2e5c1430c0 2024-03-25T18:16:12.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e620d3640 10 client.4524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:12.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e620d3640 10 monclient: _renew_subs 2024-03-25T18:16:12.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e620d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:12.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e620d3640 1 -- 172.21.15.64:0/3521717888 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e5c134970 con 0x7f2e5c1430c0 2024-03-25T18:16:12.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.395+0000 7f2e620d3640 1 librados: init done 2024-03-25T18:16:12.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.397+0000 7f2e620d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:12.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.397+0000 7f2e620d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:12.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.397+0000 7f2e620d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:12.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.397+0000 7f2e620d3640 1 -- 172.21.15.64:0/3521717888 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e5c0627b0 con 0x7f2e5c1430c0 2024-03-25T18:16:12.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.712+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e54001fe0 con 0x7f2e5c1430c0 2024-03-25T18:16:12.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.713+0000 7f2e43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:12.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.713+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2e54003b30 con 0x7f2e5c1430c0 2024-03-25T18:16:12.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.713+0000 7f2e43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:12.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.713+0000 7f2e43fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:12.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.713+0000 7f2e43fff640 10 monclient: dump: 2024-03-25T18:16:12.714 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:12.714 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:12.714 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:12.715 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:12.715 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:12.715 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:12.715 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:12.715 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:12.715 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:12.715 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:12.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.713+0000 7f2e59ffb640 4 set_mon_vals no callback set 2024-03-25T18:16:12.716 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.713+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f2e54003d50 con 0x7f2e5c1430c0 2024-03-25T18:16:12.716 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.714+0000 7f2e5b7fe640 10 monclient: get_auth_request con 0x7f2e4806de00 auth_method 0 2024-03-25T18:16:12.716 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.714+0000 7f2e43fff640 10 client.4524.objecter ms_handle_connect 0x7f2e4806de00 2024-03-25T18:16:12.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.720+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f2e540537f0 con 0x7f2e5c1430c0 2024-03-25T18:16:12.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.720+0000 7f2e43fff640 10 client.4524.objecter ms_dispatch 0x7f2e5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:12.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.720+0000 7f2e43fff640 3 client.4524.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:12.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.720+0000 7f2e43fff640 3 client.4524.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:12.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.720+0000 7f2e43fff640 20 client.4524.objecter dump_active .. 0 homeless 2024-03-25T18:16:12.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.720+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e5404f300 con 0x7f2e5c1430c0 2024-03-25T18:16:12.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.720+0000 7f2e43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:12.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.720+0000 7f2e43fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:12.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.977+0000 7f2e620d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:12.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.977+0000 7f2e620d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:12.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.977+0000 7f2e620d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:12.978 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:12.977+0000 7f2e620d3640 1 -- 172.21.15.64:0/3521717888 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e5c0b71a0 con 0x7f2e5c1430c0 2024-03-25T18:16:13.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:13.088+0000 7f2e43fff640 1 -- 172.21.15.64:0/3521717888 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f2e54051230 con 0x7f2e5c1430c0 2024-03-25T18:16:13.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:13.088+0000 7f2e43fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:13.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:13.088+0000 7f2e43fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:13.090 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:16:13.090 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:13.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:13.091+0000 7f2e620d3640 10 librados: watch_flush enter 2024-03-25T18:16:13.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:13.091+0000 7f2e620d3640 10 librados: watch_flush exit 2024-03-25T18:16:13.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:13.091+0000 7f2e620d3640 20 client.4524.objecter shutdown clearing up homeless session... 2024-03-25T18:16:13.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:13.091+0000 7f2e620d3640 10 client.4524.objecter successfully canceled tick 2024-03-25T18:16:13.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:13.091+0000 7f2e620d3640 1 -- 172.21.15.64:0/3521717888 >> v1:172.21.15.64:6800/2915543638 conn(0x7f2e4806de00 legacy=0x7f2e480702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:13.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:13.091+0000 7f2e620d3640 10 monclient: shutdown 2024-03-25T18:16:13.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:13.091+0000 7f2e620d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:13.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:13.091+0000 7f2e620d3640 1 -- 172.21.15.64:0/3521717888 >> v1:172.21.15.64:6789/0 conn(0x7f2e5c1430c0 legacy=0x7f2e5c136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:13.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:13.091+0000 7f2e620d3640 1 -- 172.21.15.64:0/3521717888 shutdown_connections 2024-03-25T18:16:13.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:13.091+0000 7f2e620d3640 1 -- 172.21.15.64:0/3521717888 >> 172.21.15.64:0/3521717888 conn(0x7f2e5c0b7ea0 msgr2=0x7f2e5c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:13.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:13.091+0000 7f2e620d3640 1 -- 172.21.15.64:0/3521717888 shutdown_connections 2024-03-25T18:16:13.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:13.091+0000 7f2e620d3640 1 -- 172.21.15.64:0/3521717888 wait complete. 2024-03-25T18:16:13.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:13.091+0000 7f2e620d3640 1 librados: shutdown 2024-03-25T18:16:13.104 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:17.614 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:07:00.726 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:17.665 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:07:00.778 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:17.694 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:07:00.807 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:17.720 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:07:00.833 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:17.725 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:07:00.804 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:17.732 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:07:00.810 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:17.739 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:07:00.851 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:17.760 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:07:00.839 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:17.774 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:07:00.887 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:19.910 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:20.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.021+0000 7fe28620d640 10 monclient: get_monmap_and_config 2024-03-25T18:16:20.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.021+0000 7fe28620d640 10 monclient: build_initial_monmap 2024-03-25T18:16:20.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.021+0000 7fe28620d640 10 monclient: monmap: 2024-03-25T18:16:20.022 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:16:20.022 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:20.022 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:16:20.022040+0000 2024-03-25T18:16:20.022 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:16:20.022040+0000 2024-03-25T18:16:20.022 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:20.022 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:20.022 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:16:20.023 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:16:20.023 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:16:20.023 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:20.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.022+0000 7fe28620d640 1 Processor -- start 2024-03-25T18:16:20.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.022+0000 7fe28620d640 1 -- start start 2024-03-25T18:16:20.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.022+0000 7fe28620d640 10 monclient: init 2024-03-25T18:16:20.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.022+0000 7fe28620d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:20.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.022+0000 7fe28620d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:20.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.022+0000 7fe28620d640 10 monclient(hunting): picked mon.noname-a con 0x7fe2800a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:16:20.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.022+0000 7fe28620d640 10 monclient(hunting): picked mon.noname-c con 0x7fe28013f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:16:20.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.023+0000 7fe28620d640 10 monclient(hunting): picked mon.noname-b con 0x7fe2801430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:16:20.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.023+0000 7fe28620d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe280142520 con 0x7fe2800a98f0 2024-03-25T18:16:20.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.023+0000 7fe28620d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2800b28a0 con 0x7fe28013f710 2024-03-25T18:16:20.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.023+0000 7fe28620d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2800b2a30 con 0x7fe2801430c0 2024-03-25T18:16:20.025 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.023+0000 7fe28620d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:20.025 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.023+0000 7fe28620d640 10 monclient(hunting): authenticate will time out at 7366.261634s 2024-03-25T18:16:20.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.028+0000 7fe27effd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe28013f710 0x7fe2801429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:51702/0 (socket says 172.21.15.64:51702) 2024-03-25T18:16:20.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.028+0000 7fe27effd640 1 -- 172.21.15.64:0/1573405363 learned_addr learned my addr 172.21.15.64:0/1573405363 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:16:20.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.036+0000 7fe27e7fc640 1 -- 172.21.15.64:0/1573405363 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644856295 0 0) 0x7fe2800b28a0 con 0x7fe28013f710 2024-03-25T18:16:20.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.036+0000 7fe27e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.036+0000 7fe27e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.036+0000 7fe27e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.036+0000 7fe27e7fc640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:16:20.038 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.036+0000 7fe27e7fc640 1 -- 172.21.15.64:0/1573405363 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe268003520 con 0x7fe28013f710 2024-03-25T18:16:20.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.043+0000 7fe27e7fc640 1 -- 172.21.15.64:0/1573405363 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3503965452 0 0) 0x7fe268003520 con 0x7fe28013f710 2024-03-25T18:16:20.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.043+0000 7fe27e7fc640 1 -- 172.21.15.64:0/1573405363 >> v1:172.21.15.64:6790/0 conn(0x7fe2801430c0 legacy=0x7fe28014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:16:20.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.043+0000 7fe27e7fc640 1 -- 172.21.15.64:0/1573405363 >> v1:172.21.15.64:6789/0 conn(0x7fe2800a98f0 legacy=0x7fe2800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.043+0000 7fe27e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:20.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.043+0000 7fe27e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:16:20.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.043+0000 7fe27e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:20.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.043+0000 7fe27e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:16:20.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.043+0000 7fe27e7fc640 1 -- 172.21.15.64:0/1573405363 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2800b44c0 con 0x7fe28013f710 2024-03-25T18:16:20.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.043+0000 7fe27e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:20.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.043+0000 7fe27e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:20.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.043+0000 7fe27e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:20.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.043+0000 7fe27e7fc640 1 -- 172.21.15.64:0/1573405363 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe274001690 con 0x7fe28013f710 2024-03-25T18:16:20.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.043+0000 7fe27e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.043+0000 7fe27e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:20.045 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.043+0000 7fe27e7fc640 10 monclient: dump: 2024-03-25T18:16:20.045 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:20.046 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:20.046 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:20.046 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:20.046 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.046 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:20.046 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:20.046 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:20.046 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:20.046 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:20.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.043+0000 7fe28620d640 5 monclient: authenticate success, global_id 4525 2024-03-25T18:16:20.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.043+0000 7fe28620d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:20.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.053+0000 7fe27e7fc640 1 -- 172.21.15.64:0/1573405363 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe274001860 con 0x7fe28013f710 2024-03-25T18:16:20.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.053+0000 7fe27e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:20.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.053+0000 7fe27e7fc640 1 -- 172.21.15.64:0/1573405363 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe274001cb0 con 0x7fe28013f710 2024-03-25T18:16:20.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.053+0000 7fe28620d640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:20.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.053+0000 7fe28620d640 4 set_mon_vals no callback set 2024-03-25T18:16:20.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.053+0000 7fe27e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.053+0000 7fe27e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:20.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.053+0000 7fe27e7fc640 10 monclient: dump: 2024-03-25T18:16:20.054 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:20.054 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:20.055 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:20.055 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:20.055 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.055 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:20.055 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:20.055 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:20.055 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:20.055 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:20.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.054+0000 7fe28620d640 10 monclient: shutdown 2024-03-25T18:16:20.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.054+0000 7fe28620d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:20.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.054+0000 7fe28620d640 1 -- 172.21.15.64:0/1573405363 >> v1:172.21.15.86:6789/0 conn(0x7fe28013f710 legacy=0x7fe2801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.054+0000 7fe28620d640 1 -- 172.21.15.64:0/1573405363 shutdown_connections 2024-03-25T18:16:20.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.054+0000 7fe28620d640 1 -- 172.21.15.64:0/1573405363 >> 172.21.15.64:0/1573405363 conn(0x7fe2800b7ea0 msgr2=0x7fe2800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:20.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.054+0000 7fe28620d640 1 -- 172.21.15.64:0/1573405363 shutdown_connections 2024-03-25T18:16:20.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.054+0000 7fe28620d640 1 -- 172.21.15.64:0/1573405363 wait complete. 2024-03-25T18:16:20.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.055+0000 7fe28620d640 10 monclient: build_initial_monmap 2024-03-25T18:16:20.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.055+0000 7fe28620d640 10 monclient: monmap: 2024-03-25T18:16:20.057 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:16:20.057 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:20.057 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:16:20.057 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:16:20.057 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:20.057 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:16:20.057 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:16:20.057 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:16:20.057 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:16:20.057 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:20.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.055+0000 7fe28620d640 1 librados: starting msgr at 2024-03-25T18:16:20.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.055+0000 7fe28620d640 1 librados: starting objecter 2024-03-25T18:16:20.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.055+0000 7fe28620d640 1 Processor -- start 2024-03-25T18:16:20.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.055+0000 7fe28620d640 1 -- start start 2024-03-25T18:16:20.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.055+0000 7fe28620d640 1 librados: setting wanted keys 2024-03-25T18:16:20.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.055+0000 7fe28620d640 1 librados: calling monclient init 2024-03-25T18:16:20.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.055+0000 7fe28620d640 10 monclient: init 2024-03-25T18:16:20.058 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.056+0000 7fe28620d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:20.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.056+0000 7fe28620d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:20.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.056+0000 7fe28620d640 10 monclient(hunting): picked mon.noname-b con 0x7fe2800a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:16:20.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.056+0000 7fe28620d640 10 monclient(hunting): picked mon.noname-c con 0x7fe28013f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:16:20.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.056+0000 7fe28620d640 10 monclient(hunting): picked mon.noname-a con 0x7fe2801430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:16:20.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.056+0000 7fe28620d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2801359a0 con 0x7fe2801430c0 2024-03-25T18:16:20.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.056+0000 7fe28620d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2801319b0 con 0x7fe2800a98f0 2024-03-25T18:16:20.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.056+0000 7fe28620d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2800b7bb0 con 0x7fe28013f710 2024-03-25T18:16:20.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.056+0000 7fe28620d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:20.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.056+0000 7fe28620d640 10 monclient(hunting): authenticate will time out at 7366.294883s 2024-03-25T18:16:20.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.074+0000 7fe27ffff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fe2801430c0 0x7fe2801312a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:48602/0 (socket says 172.21.15.64:48602) 2024-03-25T18:16:20.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.074+0000 7fe27ffff640 1 -- 172.21.15.64:0/3851564760 learned_addr learned my addr 172.21.15.64:0/3851564760 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:16:20.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.075+0000 7fe267fff640 10 client.?.objecter ms_handle_connect 0x7fe2801430c0 2024-03-25T18:16:20.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.075+0000 7fe267fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:20.078 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.077+0000 7fe267fff640 10 client.?.objecter ms_handle_connect 0x7fe28013f710 2024-03-25T18:16:20.078 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.077+0000 7fe267fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:20.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.085+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5448070 0 0) 0x7fe2801359a0 con 0x7fe2801430c0 2024-03-25T18:16:20.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.085+0000 7fe267fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.085+0000 7fe267fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.085+0000 7fe267fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.085+0000 7fe267fff640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:16:20.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.085+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe26c004eb0 con 0x7fe2801430c0 2024-03-25T18:16:20.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.087+0000 7fe267fff640 10 client.?.objecter ms_handle_connect 0x7fe2800a98f0 2024-03-25T18:16:20.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.087+0000 7fe267fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:20.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.092+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427929225 0 0) 0x7fe2801319b0 con 0x7fe2800a98f0 2024-03-25T18:16:20.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.092+0000 7fe267fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.092+0000 7fe267fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.092+0000 7fe267fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.092+0000 7fe267fff640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:16:20.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.092+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2801359a0 con 0x7fe2800a98f0 2024-03-25T18:16:20.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.094+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832375837 0 0) 0x7fe2800b7bb0 con 0x7fe28013f710 2024-03-25T18:16:20.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.094+0000 7fe267fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.094+0000 7fe267fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.094+0000 7fe267fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.094+0000 7fe267fff640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:16:20.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.094+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2801319b0 con 0x7fe28013f710 2024-03-25T18:16:20.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.099+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2175650735 0 0) 0x7fe2801359a0 con 0x7fe2800a98f0 2024-03-25T18:16:20.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.099+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2800b7bb0 con 0x7fe2800a98f0 2024-03-25T18:16:20.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.099+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe2700018b0 con 0x7fe2800a98f0 2024-03-25T18:16:20.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.099+0000 7fe267fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.099+0000 7fe267fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:20.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.099+0000 7fe267fff640 10 monclient(hunting): dump: 2024-03-25T18:16:20.101 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:20.101 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:20.101 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:20.101 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:20.101 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.102 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:20.102 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:20.102 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:20.102 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:20.102 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:20.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.099+0000 7fe267fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:20.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.103+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2050951778 0 0) 0x7fe26c004eb0 con 0x7fe2801430c0 2024-03-25T18:16:20.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.103+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2801359a0 con 0x7fe2801430c0 2024-03-25T18:16:20.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.103+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe278001ad0 con 0x7fe2801430c0 2024-03-25T18:16:20.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.103+0000 7fe267fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.103+0000 7fe267fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:20.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.103+0000 7fe267fff640 10 monclient(hunting): dump: 2024-03-25T18:16:20.104 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:20.104 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:20.105 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:20.105 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:20.105 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.105 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:20.105 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:20.105 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:20.105 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:20.105 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2158581150 0 0) 0x7fe2800b7bb0 con 0x7fe2800a98f0 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 >> v1:172.21.15.64:6790/0 conn(0x7fe28013f710 legacy=0x7fe280130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 >> v1:172.21.15.64:6789/0 conn(0x7fe2801430c0 legacy=0x7fe2801312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe267fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe267fff640 1 monclient: found mon.b 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe267fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe267fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:16:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe280135070 con 0x7fe2800a98f0 2024-03-25T18:16:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe267fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe267fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe267fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe28620d640 5 monclient: authenticate success, global_id 4528 2024-03-25T18:16:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe28620d640 10 monclient: _renew_subs 2024-03-25T18:16:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe28620d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:16:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe28620d640 1 -- 172.21.15.64:0/3851564760 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28013b400 con 0x7fe2800a98f0 2024-03-25T18:16:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe28620d640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe28620d640 10 monclient: _renew_subs 2024-03-25T18:16:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe28620d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:16:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe28620d640 1 -- 172.21.15.64:0/3851564760 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2800b75a0 con 0x7fe2800a98f0 2024-03-25T18:16:20.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.107+0000 7fe28620d640 1 librados: init done 2024-03-25T18:16:20.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.108+0000 7fe28620d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:20.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.108+0000 7fe28620d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:20.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.108+0000 7fe28620d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:16:20.110 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.108+0000 7fe28620d640 1 -- 172.21.15.64:0/3851564760 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28014f320 con 0x7fe2800a98f0 2024-03-25T18:16:20.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.394+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe270003610 con 0x7fe2800a98f0 2024-03-25T18:16:20.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.394+0000 7fe267fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:20.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.394+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe270003a60 con 0x7fe2800a98f0 2024-03-25T18:16:20.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.394+0000 7fe267fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.394+0000 7fe267fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:20.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.394+0000 7fe267fff640 10 monclient: dump: 2024-03-25T18:16:20.395 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:20.396 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:20.396 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:20.396 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:20.396 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.396 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:20.396 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:20.396 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:20.396 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:20.396 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:20.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.394+0000 7fe27dffb640 4 set_mon_vals no callback set 2024-03-25T18:16:20.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.395+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fe270003c40 con 0x7fe2800a98f0 2024-03-25T18:16:20.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.395+0000 7fe27effd640 10 monclient: get_auth_request con 0x7fe26c06df20 auth_method 0 2024-03-25T18:16:20.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.396+0000 7fe267fff640 10 client.4528.objecter ms_handle_connect 0x7fe26c06df20 2024-03-25T18:16:20.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.400+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fe2700535d0 con 0x7fe2800a98f0 2024-03-25T18:16:20.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.400+0000 7fe267fff640 10 client.4528.objecter ms_dispatch 0x7fe280000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:20.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.400+0000 7fe267fff640 3 client.4528.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:20.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.400+0000 7fe267fff640 3 client.4528.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:20.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.400+0000 7fe267fff640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-25T18:16:20.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.400+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe27004f0e0 con 0x7fe2800a98f0 2024-03-25T18:16:20.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.400+0000 7fe267fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:20.401 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.400+0000 7fe267fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:20.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.655+0000 7fe28620d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:20.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.655+0000 7fe28620d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:20.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.655+0000 7fe28620d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:16:20.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.655+0000 7fe28620d640 1 -- 172.21.15.64:0/3851564760 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe280074cc0 con 0x7fe2800a98f0 2024-03-25T18:16:20.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.685+0000 7fe267fff640 1 -- 172.21.15.64:0/3851564760 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fe270051010 con 0x7fe2800a98f0 2024-03-25T18:16:20.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.686+0000 7fe267fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:20.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.686+0000 7fe267fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:20.687 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:16:20.687 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:20.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.688+0000 7fe28620d640 10 librados: watch_flush enter 2024-03-25T18:16:20.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.688+0000 7fe28620d640 10 librados: watch_flush exit 2024-03-25T18:16:20.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.688+0000 7fe28620d640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-25T18:16:20.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.688+0000 7fe28620d640 10 client.4528.objecter successfully canceled tick 2024-03-25T18:16:20.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.688+0000 7fe28620d640 1 -- 172.21.15.64:0/3851564760 >> v1:172.21.15.64:6800/2915543638 conn(0x7fe26c06df20 legacy=0x7fe26c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.688+0000 7fe28620d640 10 monclient: shutdown 2024-03-25T18:16:20.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.688+0000 7fe28620d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:20.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.688+0000 7fe28620d640 1 -- 172.21.15.64:0/3851564760 >> v1:172.21.15.86:6789/0 conn(0x7fe2800a98f0 legacy=0x7fe2801d7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.688+0000 7fe28620d640 1 -- 172.21.15.64:0/3851564760 shutdown_connections 2024-03-25T18:16:20.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.688+0000 7fe28620d640 1 -- 172.21.15.64:0/3851564760 >> 172.21.15.64:0/3851564760 conn(0x7fe2800b7ea0 msgr2=0x7fe280074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:20.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.688+0000 7fe28620d640 1 -- 172.21.15.64:0/3851564760 shutdown_connections 2024-03-25T18:16:20.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.688+0000 7fe28620d640 1 -- 172.21.15.64:0/3851564760 wait complete. 2024-03-25T18:16:20.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:20.688+0000 7fe28620d640 1 librados: shutdown 2024-03-25T18:16:20.701 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:27.508 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:27.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.621+0000 7fe51d896640 10 monclient: get_monmap_and_config 2024-03-25T18:16:27.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.621+0000 7fe51d896640 10 monclient: build_initial_monmap 2024-03-25T18:16:27.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.621+0000 7fe51d896640 10 monclient: monmap: 2024-03-25T18:16:27.623 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:16:27.623 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:27.623 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:16:27.622787+0000 2024-03-25T18:16:27.623 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:16:27.622787+0000 2024-03-25T18:16:27.623 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:27.623 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:27.623 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:16:27.623 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:16:27.623 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:16:27.623 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:27.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.622+0000 7fe51d896640 1 Processor -- start 2024-03-25T18:16:27.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.622+0000 7fe51d896640 1 -- start start 2024-03-25T18:16:27.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.622+0000 7fe51d896640 10 monclient: init 2024-03-25T18:16:27.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.623+0000 7fe51d896640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:27.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.623+0000 7fe51d896640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:27.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.623+0000 7fe51d896640 10 monclient(hunting): picked mon.noname-a con 0x7fe5180a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:16:27.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.623+0000 7fe51d896640 10 monclient(hunting): picked mon.noname-c con 0x7fe51813f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:16:27.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.623+0000 7fe51d896640 10 monclient(hunting): picked mon.noname-b con 0x7fe5181430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:16:27.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.623+0000 7fe51d896640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe518142520 con 0x7fe5180a98f0 2024-03-25T18:16:27.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.623+0000 7fe51d896640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5180b28a0 con 0x7fe51813f710 2024-03-25T18:16:27.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.623+0000 7fe51d896640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5180b2a30 con 0x7fe5181430c0 2024-03-25T18:16:27.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.623+0000 7fe51d896640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:27.625 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.623+0000 7fe51d896640 10 monclient(hunting): authenticate will time out at 7373.862269s 2024-03-25T18:16:27.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.627+0000 7fe5177fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fe5181430c0 0x7fe51814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:55298/0 (socket says 172.21.15.64:55298) 2024-03-25T18:16:27.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.627+0000 7fe5177fe640 1 -- 172.21.15.64:0/3276403340 learned_addr learned my addr 172.21.15.64:0/3276403340 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:16:27.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.639+0000 7fe5167fc640 1 -- 172.21.15.64:0/3276403340 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726778823 0 0) 0x7fe5180b2a30 con 0x7fe5181430c0 2024-03-25T18:16:27.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.640+0000 7fe5167fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:27.640 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.640+0000 7fe5167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:27.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.640+0000 7fe5167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:27.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.640+0000 7fe5167fc640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:16:27.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.640+0000 7fe5167fc640 1 -- 172.21.15.64:0/3276403340 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe504003520 con 0x7fe5181430c0 2024-03-25T18:16:27.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.653+0000 7fe5167fc640 1 -- 172.21.15.64:0/3276403340 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467088183 0 0) 0x7fe5180b28a0 con 0x7fe51813f710 2024-03-25T18:16:27.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.653+0000 7fe5167fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:27.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.653+0000 7fe5167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:27.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.653+0000 7fe5167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:27.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.653+0000 7fe5167fc640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:16:27.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.653+0000 7fe5167fc640 1 -- 172.21.15.64:0/3276403340 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5180b2a30 con 0x7fe51813f710 2024-03-25T18:16:27.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.657+0000 7fe5167fc640 1 -- 172.21.15.64:0/3276403340 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2838605089 0 0) 0x7fe504003520 con 0x7fe5181430c0 2024-03-25T18:16:27.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.657+0000 7fe5167fc640 1 -- 172.21.15.64:0/3276403340 >> v1:172.21.15.86:6789/0 conn(0x7fe51813f710 legacy=0x7fe5181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.657+0000 7fe5167fc640 1 -- 172.21.15.64:0/3276403340 >> v1:172.21.15.64:6789/0 conn(0x7fe5180a98f0 legacy=0x7fe5180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.657+0000 7fe5167fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:27.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.657+0000 7fe5167fc640 1 monclient: found mon.noname-b 2024-03-25T18:16:27.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.657+0000 7fe5167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:27.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.657+0000 7fe5167fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:16:27.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.657+0000 7fe5167fc640 1 -- 172.21.15.64:0/3276403340 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5180b44c0 con 0x7fe5181430c0 2024-03-25T18:16:27.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.657+0000 7fe5167fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:27.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.657+0000 7fe5167fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:27.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.657+0000 7fe5167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:27.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.657+0000 7fe5167fc640 1 -- 172.21.15.64:0/3276403340 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe510001a10 con 0x7fe5181430c0 2024-03-25T18:16:27.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.657+0000 7fe5167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:27.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.657+0000 7fe5167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:27.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.657+0000 7fe5167fc640 10 monclient: dump: 2024-03-25T18:16:27.659 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:27.659 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:27.659 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:27.659 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:27.659 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:27.659 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:27.659 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:27.659 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:27.659 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:27.659 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:27.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.657+0000 7fe51d896640 5 monclient: authenticate success, global_id 4535 2024-03-25T18:16:27.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.657+0000 7fe51d896640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:27.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.690+0000 7fe5167fc640 1 -- 172.21.15.64:0/3276403340 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe510001b70 con 0x7fe5181430c0 2024-03-25T18:16:27.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.690+0000 7fe5167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:27.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.690+0000 7fe5167fc640 1 -- 172.21.15.64:0/3276403340 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe510001f80 con 0x7fe5181430c0 2024-03-25T18:16:27.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.690+0000 7fe51d896640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:27.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.690+0000 7fe51d896640 4 set_mon_vals no callback set 2024-03-25T18:16:27.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.690+0000 7fe5167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:27.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.690+0000 7fe5167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:27.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.690+0000 7fe5167fc640 10 monclient: dump: 2024-03-25T18:16:27.692 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:27.692 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:27.692 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:27.692 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:27.692 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:27.692 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:27.692 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:27.693 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:27.693 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:27.693 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:27.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.691+0000 7fe51d896640 10 monclient: shutdown 2024-03-25T18:16:27.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.691+0000 7fe51d896640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:27.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.691+0000 7fe51d896640 1 -- 172.21.15.64:0/3276403340 >> v1:172.21.15.64:6790/0 conn(0x7fe5181430c0 legacy=0x7fe51814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.692+0000 7fe51d896640 1 -- 172.21.15.64:0/3276403340 shutdown_connections 2024-03-25T18:16:27.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.692+0000 7fe51d896640 1 -- 172.21.15.64:0/3276403340 >> 172.21.15.64:0/3276403340 conn(0x7fe5180b7ea0 msgr2=0x7fe5180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:27.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.692+0000 7fe51d896640 1 -- 172.21.15.64:0/3276403340 shutdown_connections 2024-03-25T18:16:27.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.692+0000 7fe51d896640 1 -- 172.21.15.64:0/3276403340 wait complete. 2024-03-25T18:16:27.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.692+0000 7fe51d896640 10 monclient: build_initial_monmap 2024-03-25T18:16:27.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.692+0000 7fe51d896640 10 monclient: monmap: 2024-03-25T18:16:27.695 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:16:27.695 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:27.695 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:16:27.695 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:16:27.695 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:27.695 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:16:27.695 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:16:27.695 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:16:27.695 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:16:27.695 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:27.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.693+0000 7fe51d896640 1 librados: starting msgr at 2024-03-25T18:16:27.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.693+0000 7fe51d896640 1 librados: starting objecter 2024-03-25T18:16:27.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.693+0000 7fe51d896640 1 Processor -- start 2024-03-25T18:16:27.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.693+0000 7fe51d896640 1 -- start start 2024-03-25T18:16:27.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.693+0000 7fe51d896640 1 librados: setting wanted keys 2024-03-25T18:16:27.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.693+0000 7fe51d896640 1 librados: calling monclient init 2024-03-25T18:16:27.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.693+0000 7fe51d896640 10 monclient: init 2024-03-25T18:16:27.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.693+0000 7fe51d896640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:27.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.694+0000 7fe51d896640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:27.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.694+0000 7fe51d896640 10 monclient(hunting): picked mon.noname-b con 0x7fe5180a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:16:27.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.694+0000 7fe51d896640 10 monclient(hunting): picked mon.noname-a con 0x7fe51813f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:16:27.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.694+0000 7fe51d896640 10 monclient(hunting): picked mon.noname-c con 0x7fe5181430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:16:27.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.694+0000 7fe51d896640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5181396e0 con 0x7fe51813f710 2024-03-25T18:16:27.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.694+0000 7fe51d896640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe518138530 con 0x7fe5180a98f0 2024-03-25T18:16:27.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.694+0000 7fe51d896640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5181399c0 con 0x7fe5181430c0 2024-03-25T18:16:27.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.694+0000 7fe51d896640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:27.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.694+0000 7fe51d896640 10 monclient(hunting): authenticate will time out at 7373.932601s 2024-03-25T18:16:27.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.704+0000 7fe516ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe5180a98f0 0x7fe518137820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:35434/0 (socket says 172.21.15.64:35434) 2024-03-25T18:16:27.705 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.704+0000 7fe516ffd640 1 -- 172.21.15.64:0/2529728844 learned_addr learned my addr 172.21.15.64:0/2529728844 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:16:27.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.709+0000 7fe514ff9640 10 client.?.objecter ms_handle_connect 0x7fe51813f710 2024-03-25T18:16:27.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.709+0000 7fe514ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:27.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.709+0000 7fe514ff9640 10 client.?.objecter ms_handle_connect 0x7fe5181430c0 2024-03-25T18:16:27.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.709+0000 7fe514ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:27.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.709+0000 7fe514ff9640 10 client.?.objecter ms_handle_connect 0x7fe5180a98f0 2024-03-25T18:16:27.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.709+0000 7fe514ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:27.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.718+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172505755 0 0) 0x7fe5181399c0 con 0x7fe5181430c0 2024-03-25T18:16:27.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.718+0000 7fe514ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:27.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.718+0000 7fe514ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:27.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.718+0000 7fe514ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:27.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.718+0000 7fe514ff9640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:16:27.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.718+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f8004eb0 con 0x7fe5181430c0 2024-03-25T18:16:27.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.718+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713345903 0 0) 0x7fe518138530 con 0x7fe5180a98f0 2024-03-25T18:16:27.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.718+0000 7fe514ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:27.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.718+0000 7fe514ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:27.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.718+0000 7fe514ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:27.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.718+0000 7fe514ff9640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:16:27.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.718+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5181399c0 con 0x7fe5180a98f0 2024-03-25T18:16:27.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.725+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962769319 0 0) 0x7fe5181396e0 con 0x7fe51813f710 2024-03-25T18:16:27.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.725+0000 7fe514ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:27.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.725+0000 7fe514ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:27.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.725+0000 7fe514ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:27.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.725+0000 7fe514ff9640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:16:27.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.725+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe518138530 con 0x7fe51813f710 2024-03-25T18:16:27.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.732+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2090757608 0 0) 0x7fe5181399c0 con 0x7fe5180a98f0 2024-03-25T18:16:27.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.732+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5181396e0 con 0x7fe5180a98f0 2024-03-25T18:16:27.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.732+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe508001970 con 0x7fe5180a98f0 2024-03-25T18:16:27.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.732+0000 7fe514ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:27.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.732+0000 7fe514ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:27.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.732+0000 7fe514ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:27.734 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:27.734 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:27.734 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:27.734 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:27.734 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:27.734 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:27.734 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:27.734 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:27.734 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:27.734 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:27.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.732+0000 7fe514ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:27.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.738+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1235300534 0 0) 0x7fe4f8004eb0 con 0x7fe5181430c0 2024-03-25T18:16:27.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.738+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5181399c0 con 0x7fe5181430c0 2024-03-25T18:16:27.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.738+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe510001ec0 con 0x7fe5181430c0 2024-03-25T18:16:27.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.738+0000 7fe514ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:27.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.738+0000 7fe514ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:27.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.738+0000 7fe514ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:27.739 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:27.739 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:27.740 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:27.740 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:27.740 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:27.740 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:27.740 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:27.740 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:27.740 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:27.740 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:27.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.744+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3437202004 0 0) 0x7fe5181399c0 con 0x7fe5181430c0 2024-03-25T18:16:27.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.744+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 >> v1:172.21.15.86:6789/0 conn(0x7fe5180a98f0 legacy=0x7fe518137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.744+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 >> v1:172.21.15.64:6789/0 conn(0x7fe51813f710 legacy=0x7fe5181da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.744+0000 7fe514ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:16:27.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.744+0000 7fe514ff9640 1 monclient: found mon.c 2024-03-25T18:16:27.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.744+0000 7fe514ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:27.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.744+0000 7fe514ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:16:27.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.744+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe518138a50 con 0x7fe5181430c0 2024-03-25T18:16:27.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.744+0000 7fe514ff9640 10 monclient: _finish_auth 0 2024-03-25T18:16:27.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.744+0000 7fe514ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:16:27.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.744+0000 7fe514ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:27.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.745+0000 7fe51d896640 5 monclient: authenticate success, global_id 4538 2024-03-25T18:16:27.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.745+0000 7fe51d896640 10 monclient: _renew_subs 2024-03-25T18:16:27.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.745+0000 7fe51d896640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:16:27.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.745+0000 7fe51d896640 1 -- 172.21.15.64:0/2529728844 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5181380a0 con 0x7fe5181430c0 2024-03-25T18:16:27.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.745+0000 7fe51d896640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:27.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.745+0000 7fe51d896640 10 monclient: _renew_subs 2024-03-25T18:16:27.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.745+0000 7fe51d896640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:16:27.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.745+0000 7fe51d896640 1 -- 172.21.15.64:0/2529728844 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5181db140 con 0x7fe5181430c0 2024-03-25T18:16:27.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.745+0000 7fe51d896640 1 librados: init done 2024-03-25T18:16:27.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.746+0000 7fe51d896640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:27.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.746+0000 7fe51d896640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:27.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.746+0000 7fe51d896640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:16:27.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:27.746+0000 7fe51d896640 1 -- 172.21.15.64:0/2529728844 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe518138ca0 con 0x7fe5181430c0 2024-03-25T18:16:28.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.093+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5100020c0 con 0x7fe5181430c0 2024-03-25T18:16:28.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.093+0000 7fe514ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:28.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.093+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fe510004e10 con 0x7fe5181430c0 2024-03-25T18:16:28.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.093+0000 7fe514ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:28.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.093+0000 7fe514ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:28.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.093+0000 7fe514ff9640 10 monclient: dump: 2024-03-25T18:16:28.094 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:28.095 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:28.095 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:28.095 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:28.095 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:28.095 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:28.095 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:28.095 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:28.095 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:28.095 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:28.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.093+0000 7fe515ffb640 4 set_mon_vals no callback set 2024-03-25T18:16:28.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.094+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fe51001ade0 con 0x7fe5181430c0 2024-03-25T18:16:28.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.094+0000 7fe516ffd640 10 monclient: get_auth_request con 0x7fe4f806dfb0 auth_method 0 2024-03-25T18:16:28.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.094+0000 7fe514ff9640 10 client.4538.objecter ms_handle_connect 0x7fe4f806dfb0 2024-03-25T18:16:28.101 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.100+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fe5100549b0 con 0x7fe5181430c0 2024-03-25T18:16:28.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.100+0000 7fe514ff9640 10 client.4538.objecter ms_dispatch 0x7fe518000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:28.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.100+0000 7fe514ff9640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:28.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.100+0000 7fe514ff9640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:28.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.100+0000 7fe514ff9640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-25T18:16:28.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.100+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5100502f0 con 0x7fe5181430c0 2024-03-25T18:16:28.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.100+0000 7fe514ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:28.102 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.100+0000 7fe514ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:28.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.328+0000 7fe51d896640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:28.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.328+0000 7fe51d896640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:28.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.328+0000 7fe51d896640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:16:28.329 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.328+0000 7fe51d896640 1 -- 172.21.15.64:0/2529728844 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5181386c0 con 0x7fe5181430c0 2024-03-25T18:16:28.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.418+0000 7fe514ff9640 1 -- 172.21.15.64:0/2529728844 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fe5100523f0 con 0x7fe5181430c0 2024-03-25T18:16:28.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.418+0000 7fe514ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:28.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.418+0000 7fe514ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:28.420 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:16:28.420 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:28.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.421+0000 7fe51d896640 10 librados: watch_flush enter 2024-03-25T18:16:28.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.421+0000 7fe51d896640 10 librados: watch_flush exit 2024-03-25T18:16:28.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.421+0000 7fe51d896640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-25T18:16:28.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.421+0000 7fe51d896640 10 client.4538.objecter successfully canceled tick 2024-03-25T18:16:28.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.421+0000 7fe51d896640 1 -- 172.21.15.64:0/2529728844 >> v1:172.21.15.64:6800/2915543638 conn(0x7fe4f806dfb0 legacy=0x7fe4f8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:28.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.421+0000 7fe51d896640 10 monclient: shutdown 2024-03-25T18:16:28.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.421+0000 7fe51d896640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:28.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.421+0000 7fe51d896640 1 -- 172.21.15.64:0/2529728844 >> v1:172.21.15.64:6790/0 conn(0x7fe5181430c0 legacy=0x7fe5181e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:28.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.421+0000 7fe51d896640 1 -- 172.21.15.64:0/2529728844 shutdown_connections 2024-03-25T18:16:28.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.421+0000 7fe51d896640 1 -- 172.21.15.64:0/2529728844 >> 172.21.15.64:0/2529728844 conn(0x7fe5180b7ea0 msgr2=0x7fe518074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:28.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.422+0000 7fe51d896640 1 -- 172.21.15.64:0/2529728844 shutdown_connections 2024-03-25T18:16:28.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.422+0000 7fe51d896640 1 -- 172.21.15.64:0/2529728844 wait complete. 2024-03-25T18:16:28.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:28.422+0000 7fe51d896640 1 librados: shutdown 2024-03-25T18:16:28.433 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:35.240 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:35.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.355+0000 7f4e136f1640 10 monclient: get_monmap_and_config 2024-03-25T18:16:35.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.355+0000 7f4e136f1640 10 monclient: build_initial_monmap 2024-03-25T18:16:35.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.355+0000 7f4e136f1640 10 monclient: monmap: 2024-03-25T18:16:35.357 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:16:35.357 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:35.357 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:16:35.356162+0000 2024-03-25T18:16:35.357 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:16:35.356162+0000 2024-03-25T18:16:35.357 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:35.357 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:35.357 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:16:35.357 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:16:35.358 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:16:35.358 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:35.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.356+0000 7f4e136f1640 1 Processor -- start 2024-03-25T18:16:35.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.356+0000 7f4e136f1640 1 -- start start 2024-03-25T18:16:35.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.356+0000 7f4e136f1640 10 monclient: init 2024-03-25T18:16:35.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.356+0000 7f4e136f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:35.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.356+0000 7f4e136f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:35.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.356+0000 7f4e136f1640 10 monclient(hunting): picked mon.noname-b con 0x7f4e0c0a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:16:35.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.357+0000 7f4e136f1640 10 monclient(hunting): picked mon.noname-c con 0x7f4e0c13f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:16:35.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.357+0000 7f4e136f1640 10 monclient(hunting): picked mon.noname-a con 0x7f4e0c1430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:16:35.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.357+0000 7f4e136f1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c142520 con 0x7f4e0c1430c0 2024-03-25T18:16:35.359 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.357+0000 7f4e136f1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c0b28a0 con 0x7f4e0c13f710 2024-03-25T18:16:35.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.357+0000 7f4e136f1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c0b2a30 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.357+0000 7f4e136f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:35.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.357+0000 7f4e136f1640 10 monclient(hunting): authenticate will time out at 7381.595521s 2024-03-25T18:16:35.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.372+0000 7f4e1148d640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f4e0c0a98f0 0x7f4e0c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:60142/0 (socket says 172.21.15.64:60142) 2024-03-25T18:16:35.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.372+0000 7f4e1148d640 1 -- 172.21.15.64:0/96199671 learned_addr learned my addr 172.21.15.64:0/96199671 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:16:35.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.388+0000 7f4e03fff640 1 -- 172.21.15.64:0/96199671 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368681436 0 0) 0x7f4e0c0b28a0 con 0x7f4e0c13f710 2024-03-25T18:16:35.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.388+0000 7f4e03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:35.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.388+0000 7f4e03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:35.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.388+0000 7f4e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:35.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.388+0000 7f4e03fff640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:16:35.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.388+0000 7f4e03fff640 1 -- 172.21.15.64:0/96199671 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df8003520 con 0x7f4e0c13f710 2024-03-25T18:16:35.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.394+0000 7f4e03fff640 1 -- 172.21.15.64:0/96199671 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729217110 0 0) 0x7f4e0c142520 con 0x7f4e0c1430c0 2024-03-25T18:16:35.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.394+0000 7f4e03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:35.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.394+0000 7f4e03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:35.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.394+0000 7f4e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:35.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.394+0000 7f4e03fff640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:16:35.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.394+0000 7f4e03fff640 1 -- 172.21.15.64:0/96199671 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e0c0b28a0 con 0x7f4e0c1430c0 2024-03-25T18:16:35.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.395+0000 7f4e03fff640 1 -- 172.21.15.64:0/96199671 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73821193 0 0) 0x7f4e0c0b2a30 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.395+0000 7f4e03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:35.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.395+0000 7f4e03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:35.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.395+0000 7f4e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:35.396 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.395+0000 7f4e03fff640 10 monclient(hunting): my global_id is 4541 2024-03-25T18:16:35.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.395+0000 7f4e03fff640 1 -- 172.21.15.64:0/96199671 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e0c142520 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.403+0000 7f4e03fff640 1 -- 172.21.15.64:0/96199671 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 293994240 0 0) 0x7f4e0c0b28a0 con 0x7f4e0c1430c0 2024-03-25T18:16:35.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.403+0000 7f4e03fff640 1 -- 172.21.15.64:0/96199671 >> v1:172.21.15.64:6790/0 conn(0x7f4e0c0a98f0 legacy=0x7f4e0c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:35.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.403+0000 7f4e03fff640 1 -- 172.21.15.64:0/96199671 >> v1:172.21.15.86:6789/0 conn(0x7f4e0c13f710 legacy=0x7f4e0c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:35.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.403+0000 7f4e03fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:35.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.403+0000 7f4e03fff640 1 monclient: found mon.noname-a 2024-03-25T18:16:35.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.403+0000 7f4e03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:35.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.403+0000 7f4e03fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:16:35.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.403+0000 7f4e03fff640 1 -- 172.21.15.64:0/96199671 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0c0b44c0 con 0x7f4e0c1430c0 2024-03-25T18:16:35.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.403+0000 7f4e03fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:35.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.403+0000 7f4e03fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:35.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.403+0000 7f4e03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:35.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.403+0000 7f4e03fff640 1 -- 172.21.15.64:0/96199671 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4e040019f0 con 0x7f4e0c1430c0 2024-03-25T18:16:35.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.403+0000 7f4e136f1640 5 monclient: authenticate success, global_id 4539 2024-03-25T18:16:35.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.403+0000 7f4e136f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:35.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.403+0000 7f4e03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:35.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.403+0000 7f4e03fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:35.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.403+0000 7f4e03fff640 10 monclient: dump: 2024-03-25T18:16:35.405 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:35.406 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:35.406 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:35.406 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:35.406 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:35.406 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:35.406 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:35.406 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:35.406 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:35.406 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:35.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.403+0000 7f4e136f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:35.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.410+0000 7f4e03fff640 1 -- 172.21.15.64:0/96199671 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e04001b50 con 0x7f4e0c1430c0 2024-03-25T18:16:35.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.410+0000 7f4e03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:35.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.410+0000 7f4e03fff640 1 -- 172.21.15.64:0/96199671 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4e04001f60 con 0x7f4e0c1430c0 2024-03-25T18:16:35.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.410+0000 7f4e136f1640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:35.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.410+0000 7f4e136f1640 4 set_mon_vals no callback set 2024-03-25T18:16:35.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.410+0000 7f4e03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:35.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.410+0000 7f4e03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:35.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.410+0000 7f4e03fff640 10 monclient: dump: 2024-03-25T18:16:35.412 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:35.412 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:35.412 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:35.412 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:35.412 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:35.412 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:35.412 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:35.412 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:35.412 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:35.412 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:35.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.411+0000 7f4e136f1640 10 monclient: shutdown 2024-03-25T18:16:35.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.411+0000 7f4e136f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:35.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.411+0000 7f4e136f1640 1 -- 172.21.15.64:0/96199671 >> v1:172.21.15.64:6789/0 conn(0x7f4e0c1430c0 legacy=0x7f4e0c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:35.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.411+0000 7f4e136f1640 1 -- 172.21.15.64:0/96199671 shutdown_connections 2024-03-25T18:16:35.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.411+0000 7f4e136f1640 1 -- 172.21.15.64:0/96199671 >> 172.21.15.64:0/96199671 conn(0x7f4e0c0b7ea0 msgr2=0x7f4e0c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:35.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.411+0000 7f4e136f1640 1 -- 172.21.15.64:0/96199671 shutdown_connections 2024-03-25T18:16:35.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.411+0000 7f4e136f1640 1 -- 172.21.15.64:0/96199671 wait complete. 2024-03-25T18:16:35.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.412+0000 7f4e136f1640 10 monclient: build_initial_monmap 2024-03-25T18:16:35.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.412+0000 7f4e136f1640 10 monclient: monmap: 2024-03-25T18:16:35.414 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:16:35.414 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:35.414 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:16:35.414 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:16:35.414 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:35.414 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:16:35.414 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:16:35.414 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:16:35.414 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:16:35.414 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:35.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.413+0000 7f4e136f1640 1 librados: starting msgr at 2024-03-25T18:16:35.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.413+0000 7f4e136f1640 1 librados: starting objecter 2024-03-25T18:16:35.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.413+0000 7f4e136f1640 1 Processor -- start 2024-03-25T18:16:35.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.413+0000 7f4e136f1640 1 -- start start 2024-03-25T18:16:35.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.413+0000 7f4e136f1640 1 librados: setting wanted keys 2024-03-25T18:16:35.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.413+0000 7f4e136f1640 1 librados: calling monclient init 2024-03-25T18:16:35.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.413+0000 7f4e136f1640 10 monclient: init 2024-03-25T18:16:35.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.413+0000 7f4e136f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:35.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.413+0000 7f4e136f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:35.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.413+0000 7f4e136f1640 10 monclient(hunting): picked mon.noname-b con 0x7f4e0c0a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:16:35.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.414+0000 7f4e136f1640 10 monclient(hunting): picked mon.noname-a con 0x7f4e0c13f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:16:35.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.414+0000 7f4e136f1640 10 monclient(hunting): picked mon.noname-c con 0x7f4e0c1430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:16:35.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.414+0000 7f4e136f1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c135bf0 con 0x7f4e0c13f710 2024-03-25T18:16:35.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.414+0000 7f4e136f1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c0b6d10 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.414+0000 7f4e136f1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c1317f0 con 0x7f4e0c1430c0 2024-03-25T18:16:35.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.414+0000 7f4e136f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:35.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.414+0000 7f4e136f1640 10 monclient(hunting): authenticate will time out at 7381.652659s 2024-03-25T18:16:35.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.419+0000 7f4e1148d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4e0c0a98f0 0x7f4e0c1310e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:40254/0 (socket says 172.21.15.64:40254) 2024-03-25T18:16:35.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.419+0000 7f4e1148d640 1 -- 172.21.15.64:0/3437183508 learned_addr learned my addr 172.21.15.64:0/3437183508 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:16:35.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.421+0000 7f4e01ffb640 10 client.?.objecter ms_handle_connect 0x7f4e0c0a98f0 2024-03-25T18:16:35.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.421+0000 7f4e01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:35.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.430+0000 7f4e01ffb640 1 -- 172.21.15.64:0/3437183508 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866353013 0 0) 0x7f4e0c0b6d10 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.430+0000 7f4e01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:35.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.430+0000 7f4e01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:35.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.430+0000 7f4e01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:35.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.430+0000 7f4e01ffb640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:16:35.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.430+0000 7f4e01ffb640 1 -- 172.21.15.64:0/3437183508 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df0004eb0 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.442+0000 7f4e01ffb640 10 client.?.objecter ms_handle_connect 0x7f4e0c13f710 2024-03-25T18:16:35.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.442+0000 7f4e01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:35.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.445+0000 7f4e01ffb640 1 -- 172.21.15.64:0/3437183508 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1752382769 0 0) 0x7f4df0004eb0 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.445+0000 7f4e01ffb640 1 -- 172.21.15.64:0/3437183508 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e0c0b6d10 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.445+0000 7f4e01ffb640 1 -- 172.21.15.64:0/3437183508 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4e08001910 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.445+0000 7f4e01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:35.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.445+0000 7f4e01ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:35.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.445+0000 7f4e01ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:35.446 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:35.446 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:35.446 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:35.446 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:35.447 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:35.447 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:35.447 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:35.447 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:35.447 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:35.447 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:35.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.445+0000 7f4e01ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:35.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.451+0000 7f4e01ffb640 1 -- 172.21.15.64:0/3437183508 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1084735581 0 0) 0x7f4e0c0b6d10 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.451+0000 7f4e01ffb640 1 -- 172.21.15.64:0/3437183508 >> v1:172.21.15.64:6790/0 conn(0x7f4e0c1430c0 legacy=0x7f4e0c1db370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:35.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.451+0000 7f4e01ffb640 1 -- 172.21.15.64:0/3437183508 >> v1:172.21.15.64:6789/0 conn(0x7f4e0c13f710 legacy=0x7f4e0c1dac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:35.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.451+0000 7f4e01ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:35.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.451+0000 7f4e01ffb640 1 monclient: found mon.b 2024-03-25T18:16:35.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.451+0000 7f4e01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:35.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.451+0000 7f4e01ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:16:35.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.451+0000 7f4e01ffb640 1 -- 172.21.15.64:0/3437183508 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0c131980 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.451+0000 7f4e01ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:35.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.451+0000 7f4e01ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:35.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.451+0000 7f4e01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:35.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.451+0000 7f4e136f1640 5 monclient: authenticate success, global_id 4540 2024-03-25T18:16:35.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.451+0000 7f4e136f1640 10 monclient: _renew_subs 2024-03-25T18:16:35.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.451+0000 7f4e136f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:16:35.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.451+0000 7f4e136f1640 1 -- 172.21.15.64:0/3437183508 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e0c0b76e0 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.452+0000 7f4e136f1640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:35.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.452+0000 7f4e136f1640 10 monclient: _renew_subs 2024-03-25T18:16:35.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.452+0000 7f4e136f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:16:35.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.452+0000 7f4e136f1640 1 -- 172.21.15.64:0/3437183508 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e0c135270 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.452+0000 7f4e136f1640 1 librados: init done 2024-03-25T18:16:35.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.453+0000 7f4e136f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:35.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.453+0000 7f4e136f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:35.454 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.453+0000 7f4e136f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:16:35.455 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.453+0000 7f4e136f1640 1 -- 172.21.15.64:0/3437183508 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e0c0627b0 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.463 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.463+0000 7f4e01ffb640 1 -- 172.21.15.64:0/3437183508 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e08001a70 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.463+0000 7f4e01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:35.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.463+0000 7f4e01ffb640 1 -- 172.21.15.64:0/3437183508 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4e08004a40 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.463+0000 7f4e01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:35.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.463+0000 7f4e01ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:35.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.463+0000 7f4e01ffb640 10 monclient: dump: 2024-03-25T18:16:35.464 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:35.464 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:35.464 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:35.464 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:35.464 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:35.464 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:35.464 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:35.464 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:35.464 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:35.465 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:35.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.463+0000 7f4e037fe640 4 set_mon_vals no callback set 2024-03-25T18:16:35.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.744+0000 7f4e01ffb640 1 -- 172.21.15.64:0/3437183508 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f4e08004c20 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.745+0000 7f4e10c8c640 10 monclient: get_auth_request con 0x7f4df006dbc0 auth_method 0 2024-03-25T18:16:35.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.745+0000 7f4e01ffb640 10 client.4540.objecter ms_handle_connect 0x7f4df006dbc0 2024-03-25T18:16:35.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.747+0000 7f4e01ffb640 1 -- 172.21.15.64:0/3437183508 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f4e080549a0 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.747+0000 7f4e01ffb640 10 client.4540.objecter ms_dispatch 0x7f4e0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:35.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.747+0000 7f4e01ffb640 3 client.4540.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:35.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.747+0000 7f4e01ffb640 3 client.4540.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:35.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.747+0000 7f4e01ffb640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-25T18:16:35.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.750+0000 7f4e01ffb640 1 -- 172.21.15.64:0/3437183508 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e080502e0 con 0x7f4e0c0a98f0 2024-03-25T18:16:35.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.750+0000 7f4e01ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:35.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.750+0000 7f4e01ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:35.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.997+0000 7f4e136f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:35.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.998+0000 7f4e136f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:35.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.998+0000 7f4e136f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:16:35.999 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:35.998+0000 7f4e136f1640 1 -- 172.21.15.64:0/3437183508 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e0c13e100 con 0x7f4e0c0a98f0 2024-03-25T18:16:36.134 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:36.133+0000 7f4e01ffb640 1 -- 172.21.15.64:0/3437183508 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f4e080523e0 con 0x7f4e0c0a98f0 2024-03-25T18:16:36.134 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:36.133+0000 7f4e01ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:36.134 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:36.133+0000 7f4e01ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:36.134 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:16:36.134 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:36.136 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:36.135+0000 7f4e136f1640 10 librados: watch_flush enter 2024-03-25T18:16:36.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:36.135+0000 7f4e136f1640 10 librados: watch_flush exit 2024-03-25T18:16:36.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:36.135+0000 7f4e136f1640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-25T18:16:36.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:36.135+0000 7f4e136f1640 10 client.4540.objecter successfully canceled tick 2024-03-25T18:16:36.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:36.135+0000 7f4e136f1640 1 -- 172.21.15.64:0/3437183508 >> v1:172.21.15.64:6800/2915543638 conn(0x7f4df006dbc0 legacy=0x7f4df0070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:36.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:36.135+0000 7f4e136f1640 10 monclient: shutdown 2024-03-25T18:16:36.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:36.135+0000 7f4e136f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:36.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:36.135+0000 7f4e136f1640 1 -- 172.21.15.64:0/3437183508 >> v1:172.21.15.86:6789/0 conn(0x7f4e0c0a98f0 legacy=0x7f4e0c1310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:36.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:36.135+0000 7f4e136f1640 1 -- 172.21.15.64:0/3437183508 shutdown_connections 2024-03-25T18:16:36.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:36.135+0000 7f4e136f1640 1 -- 172.21.15.64:0/3437183508 >> 172.21.15.64:0/3437183508 conn(0x7f4e0c0b7ea0 msgr2=0x7f4e0c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:36.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:36.135+0000 7f4e136f1640 1 -- 172.21.15.64:0/3437183508 shutdown_connections 2024-03-25T18:16:36.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:36.135+0000 7f4e136f1640 1 -- 172.21.15.64:0/3437183508 wait complete. 2024-03-25T18:16:36.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:36.136+0000 7f4e136f1640 1 librados: shutdown 2024-03-25T18:16:36.147 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:37.684 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:07:20.797 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:37.732 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:07:20.845 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:37.734 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:07:20.847 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:37.744 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:07:20.857 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:37.776 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:07:20.888 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:37.788 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:07:20.901 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:37.820 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:07:20.899 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:37.824 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:07:20.903 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:37.847 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:07:20.926 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:42.954 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:43.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.053+0000 7f4be3f15640 10 monclient: get_monmap_and_config 2024-03-25T18:16:43.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.053+0000 7f4be3f15640 10 monclient: build_initial_monmap 2024-03-25T18:16:43.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.053+0000 7f4be3f15640 10 monclient: monmap: 2024-03-25T18:16:43.054 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:16:43.054 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:43.055 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:16:43.054065+0000 2024-03-25T18:16:43.055 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:16:43.054065+0000 2024-03-25T18:16:43.055 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:43.055 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:43.055 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:16:43.055 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:16:43.055 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:16:43.055 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:43.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.053+0000 7f4be3f15640 1 Processor -- start 2024-03-25T18:16:43.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.054+0000 7f4be3f15640 1 -- start start 2024-03-25T18:16:43.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.054+0000 7f4be3f15640 10 monclient: init 2024-03-25T18:16:43.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.054+0000 7f4be3f15640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:43.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.054+0000 7f4be3f15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:43.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.054+0000 7f4be3f15640 10 monclient(hunting): picked mon.noname-c con 0x7f4bdc0a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:16:43.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.054+0000 7f4be3f15640 10 monclient(hunting): picked mon.noname-b con 0x7f4bdc13f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:16:43.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.054+0000 7f4be3f15640 10 monclient(hunting): picked mon.noname-a con 0x7f4bdc1430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:16:43.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.054+0000 7f4be3f15640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc142520 con 0x7f4bdc1430c0 2024-03-25T18:16:43.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.054+0000 7f4be3f15640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc0b28a0 con 0x7f4bdc0a98f0 2024-03-25T18:16:43.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.054+0000 7f4be3f15640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc0b2a30 con 0x7f4bdc13f710 2024-03-25T18:16:43.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.054+0000 7f4be3f15640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:43.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.054+0000 7f4be3f15640 10 monclient(hunting): authenticate will time out at 7389.293194s 2024-03-25T18:16:43.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.069+0000 7f4be24b2640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f4bdc1430c0 0x7f4bdc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:46558/0 (socket says 172.21.15.64:46558) 2024-03-25T18:16:43.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.069+0000 7f4be24b2640 1 -- 172.21.15.64:0/3462333999 learned_addr learned my addr 172.21.15.64:0/3462333999 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:16:43.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.084+0000 7f4be0caf640 1 -- 172.21.15.64:0/3462333999 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307049421 0 0) 0x7f4bdc142520 con 0x7f4bdc1430c0 2024-03-25T18:16:43.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.084+0000 7f4be0caf640 10 monclient(hunting): _negotiate 2024-03-25T18:16:43.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.084+0000 7f4be0caf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:43.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.084+0000 7f4be0caf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:43.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.084+0000 7f4be0caf640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:16:43.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.084+0000 7f4be0caf640 1 -- 172.21.15.64:0/3462333999 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc8003520 con 0x7f4bdc1430c0 2024-03-25T18:16:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.087+0000 7f4be0caf640 1 -- 172.21.15.64:0/3462333999 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100775994 0 0) 0x7f4bdc0b2a30 con 0x7f4bdc13f710 2024-03-25T18:16:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.087+0000 7f4be0caf640 10 monclient(hunting): _negotiate 2024-03-25T18:16:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.087+0000 7f4be0caf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.087+0000 7f4be0caf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.087+0000 7f4be0caf640 10 monclient(hunting): my global_id is 4544 2024-03-25T18:16:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.087+0000 7f4be0caf640 1 -- 172.21.15.64:0/3462333999 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bdc142520 con 0x7f4bdc13f710 2024-03-25T18:16:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.088+0000 7f4be0caf640 1 -- 172.21.15.64:0/3462333999 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846386097 0 0) 0x7f4bdc0b28a0 con 0x7f4bdc0a98f0 2024-03-25T18:16:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.088+0000 7f4be0caf640 10 monclient(hunting): _negotiate 2024-03-25T18:16:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.088+0000 7f4be0caf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.088+0000 7f4be0caf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.088+0000 7f4be0caf640 10 monclient(hunting): my global_id is 4543 2024-03-25T18:16:43.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.088+0000 7f4be0caf640 1 -- 172.21.15.64:0/3462333999 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bdc0b2a30 con 0x7f4bdc0a98f0 2024-03-25T18:16:43.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.096+0000 7f4be0caf640 1 -- 172.21.15.64:0/3462333999 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1127320194 0 0) 0x7f4bdc0b2a30 con 0x7f4bdc0a98f0 2024-03-25T18:16:43.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.096+0000 7f4be0caf640 1 -- 172.21.15.64:0/3462333999 >> v1:172.21.15.64:6790/0 conn(0x7f4bdc13f710 legacy=0x7f4bdc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.096+0000 7f4be0caf640 1 -- 172.21.15.64:0/3462333999 >> v1:172.21.15.64:6789/0 conn(0x7f4bdc1430c0 legacy=0x7f4bdc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.096+0000 7f4be0caf640 10 monclient: _finish_hunting 0 2024-03-25T18:16:43.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.096+0000 7f4be0caf640 1 monclient: found mon.noname-c 2024-03-25T18:16:43.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.096+0000 7f4be0caf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:43.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.096+0000 7f4be0caf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:16:43.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.096+0000 7f4be0caf640 1 -- 172.21.15.64:0/3462333999 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc0b44c0 con 0x7f4bdc0a98f0 2024-03-25T18:16:43.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.096+0000 7f4be0caf640 10 monclient: _finish_auth 0 2024-03-25T18:16:43.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.096+0000 7f4be0caf640 10 monclient: _check_auth_tickets 2024-03-25T18:16:43.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.096+0000 7f4be0caf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:43.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.096+0000 7f4be0caf640 1 -- 172.21.15.64:0/3462333999 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4bd80015b0 con 0x7f4bdc0a98f0 2024-03-25T18:16:43.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.096+0000 7f4be0caf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:43.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.096+0000 7f4be0caf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:43.098 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.096+0000 7f4be0caf640 10 monclient: dump: 2024-03-25T18:16:43.098 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:43.098 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:43.098 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:43.098 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:43.098 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:43.098 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:43.099 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:43.099 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:43.099 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:43.099 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:43.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.096+0000 7f4be3f15640 5 monclient: authenticate success, global_id 4543 2024-03-25T18:16:43.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.096+0000 7f4be3f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:43.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.111+0000 7f4be0caf640 1 -- 172.21.15.64:0/3462333999 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bd8001710 con 0x7f4bdc0a98f0 2024-03-25T18:16:43.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.111+0000 7f4be0caf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:43.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.111+0000 7f4be0caf640 1 -- 172.21.15.64:0/3462333999 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4bd8001b20 con 0x7f4bdc0a98f0 2024-03-25T18:16:43.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.111+0000 7f4be3f15640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:43.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.111+0000 7f4be3f15640 4 set_mon_vals no callback set 2024-03-25T18:16:43.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.111+0000 7f4be0caf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:43.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.111+0000 7f4be0caf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:43.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.111+0000 7f4be0caf640 10 monclient: dump: 2024-03-25T18:16:43.113 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:43.113 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:43.113 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:43.113 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:43.113 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:43.113 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:43.113 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:43.113 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:43.113 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:43.113 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:43.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.112+0000 7f4be3f15640 10 monclient: shutdown 2024-03-25T18:16:43.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.112+0000 7f4be3f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:43.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.112+0000 7f4be3f15640 1 -- 172.21.15.64:0/3462333999 >> v1:172.21.15.86:6789/0 conn(0x7f4bdc0a98f0 legacy=0x7f4bdc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.112+0000 7f4be3f15640 1 -- 172.21.15.64:0/3462333999 shutdown_connections 2024-03-25T18:16:43.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.112+0000 7f4be3f15640 1 -- 172.21.15.64:0/3462333999 >> 172.21.15.64:0/3462333999 conn(0x7f4bdc0b7ea0 msgr2=0x7f4bdc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:43.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.112+0000 7f4be3f15640 1 -- 172.21.15.64:0/3462333999 shutdown_connections 2024-03-25T18:16:43.115 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.113+0000 7f4be3f15640 1 -- 172.21.15.64:0/3462333999 wait complete. 2024-03-25T18:16:43.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.114+0000 7f4be3f15640 10 monclient: build_initial_monmap 2024-03-25T18:16:43.116 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.114+0000 7f4be3f15640 10 monclient: monmap: 2024-03-25T18:16:43.116 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:16:43.117 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:43.117 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:16:43.117 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:16:43.117 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:43.117 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:16:43.117 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:16:43.117 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:16:43.117 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:16:43.117 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:43.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.114+0000 7f4be3f15640 1 librados: starting msgr at 2024-03-25T18:16:43.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.114+0000 7f4be3f15640 1 librados: starting objecter 2024-03-25T18:16:43.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.114+0000 7f4be3f15640 1 Processor -- start 2024-03-25T18:16:43.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.114+0000 7f4be3f15640 1 -- start start 2024-03-25T18:16:43.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.114+0000 7f4be3f15640 1 librados: setting wanted keys 2024-03-25T18:16:43.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.114+0000 7f4be3f15640 1 librados: calling monclient init 2024-03-25T18:16:43.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.114+0000 7f4be3f15640 10 monclient: init 2024-03-25T18:16:43.117 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.115+0000 7f4be3f15640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:43.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.115+0000 7f4be3f15640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:43.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.115+0000 7f4be3f15640 10 monclient(hunting): picked mon.noname-a con 0x7f4bdc0a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:16:43.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.115+0000 7f4be3f15640 10 monclient(hunting): picked mon.noname-b con 0x7f4bdc13f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:16:43.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.115+0000 7f4be3f15640 10 monclient(hunting): picked mon.noname-c con 0x7f4bdc1430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:16:43.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.115+0000 7f4be3f15640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc1395e0 con 0x7f4bdc0a98f0 2024-03-25T18:16:43.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.115+0000 7f4be3f15640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc1398c0 con 0x7f4bdc13f710 2024-03-25T18:16:43.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.115+0000 7f4be3f15640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc0abb00 con 0x7f4bdc1430c0 2024-03-25T18:16:43.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.115+0000 7f4be3f15640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:43.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.115+0000 7f4be3f15640 10 monclient(hunting): authenticate will time out at 7389.354072s 2024-03-25T18:16:43.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.121+0000 7f4be24b2640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f4bdc1430c0 0x7f4bdc1e2bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:57650/0 (socket says 172.21.15.64:57650) 2024-03-25T18:16:43.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.121+0000 7f4be24b2640 1 -- 172.21.15.64:0/3375791893 learned_addr learned my addr 172.21.15.64:0/3375791893 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:16:43.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.121+0000 7f4bd27fc640 10 client.?.objecter ms_handle_connect 0x7f4bdc1430c0 2024-03-25T18:16:43.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.121+0000 7f4bd27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:43.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.134+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258290785 0 0) 0x7f4bdc0abb00 con 0x7f4bdc1430c0 2024-03-25T18:16:43.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.134+0000 7f4bd27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:43.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.134+0000 7f4bd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:43.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.134+0000 7f4bd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:43.138 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.134+0000 7f4bd27fc640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:16:43.138 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.134+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc0004eb0 con 0x7f4bdc1430c0 2024-03-25T18:16:43.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.139+0000 7f4bd27fc640 10 client.?.objecter ms_handle_connect 0x7f4bdc13f710 2024-03-25T18:16:43.140 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.139+0000 7f4bd27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:43.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.141+0000 7f4bd27fc640 10 client.?.objecter ms_handle_connect 0x7f4bdc0a98f0 2024-03-25T18:16:43.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.141+0000 7f4bd27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:43.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.145+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3131083554 0 0) 0x7f4bc0004eb0 con 0x7f4bdc1430c0 2024-03-25T18:16:43.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.145+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bdc0abb00 con 0x7f4bdc1430c0 2024-03-25T18:16:43.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.145+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4bd4001b70 con 0x7f4bdc1430c0 2024-03-25T18:16:43.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.145+0000 7f4bd27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:43.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.145+0000 7f4bd27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:43.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.145+0000 7f4bd27fc640 10 monclient(hunting): dump: 2024-03-25T18:16:43.146 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:43.146 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:43.146 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:43.146 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:43.147 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:43.147 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:43.147 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:43.147 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:43.147 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:43.147 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:43.147 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.145+0000 7f4bd27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.153+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306092455 0 0) 0x7f4bdc1395e0 con 0x7f4bdc0a98f0 2024-03-25T18:16:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.153+0000 7f4bd27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.153+0000 7f4bd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.153+0000 7f4bd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.153+0000 7f4bd27fc640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:16:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.153+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc0004eb0 con 0x7f4bdc0a98f0 2024-03-25T18:16:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.153+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138945410 0 0) 0x7f4bdc1398c0 con 0x7f4bdc13f710 2024-03-25T18:16:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.153+0000 7f4bd27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:43.154 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.153+0000 7f4bd27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:43.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.153+0000 7f4bd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:43.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.153+0000 7f4bd27fc640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:16:43.155 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.153+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bdc1395e0 con 0x7f4bdc13f710 2024-03-25T18:16:43.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2302209776 0 0) 0x7f4bdc0abb00 con 0x7f4bdc1430c0 2024-03-25T18:16:43.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 >> v1:172.21.15.86:6789/0 conn(0x7f4bdc13f710 legacy=0x7f4bdc134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 >> v1:172.21.15.64:6789/0 conn(0x7f4bdc0a98f0 legacy=0x7f4bdc1daf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4bd27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4bd27fc640 1 monclient: found mon.c 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4bd27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4bd27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc137ce0 con 0x7f4bdc1430c0 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4bd27fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4bd27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4bd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4be3f15640 5 monclient: authenticate success, global_id 4547 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4be3f15640 10 monclient: _renew_subs 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4be3f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4be3f15640 1 -- 172.21.15.64:0/3375791893 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bdc1db630 con 0x7f4bdc1430c0 2024-03-25T18:16:43.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4be3f15640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:43.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4be3f15640 10 monclient: _renew_subs 2024-03-25T18:16:43.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4be3f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:16:43.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4be3f15640 1 -- 172.21.15.64:0/3375791893 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bdc134a00 con 0x7f4bdc1430c0 2024-03-25T18:16:43.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.156+0000 7f4be3f15640 1 librados: init done 2024-03-25T18:16:43.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.157+0000 7f4be3f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:43.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.157+0000 7f4be3f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:43.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.157+0000 7f4be3f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:16:43.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.157+0000 7f4be3f15640 1 -- 172.21.15.64:0/3375791893 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bdc0b3ef0 con 0x7f4bdc1430c0 2024-03-25T18:16:43.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.161+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bd40020c0 con 0x7f4bdc1430c0 2024-03-25T18:16:43.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.161+0000 7f4bd27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:43.162 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.161+0000 7f4bd3fff640 4 set_mon_vals no callback set 2024-03-25T18:16:43.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.470+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4bd4004d40 con 0x7f4bdc1430c0 2024-03-25T18:16:43.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.470+0000 7f4bd27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:43.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.470+0000 7f4bd27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:43.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.470+0000 7f4bd27fc640 10 monclient: dump: 2024-03-25T18:16:43.472 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:43.472 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:43.472 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:43.472 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:43.472 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:43.472 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:43.472 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:43.473 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:43.473 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:43.473 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:43.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.470+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f4bd401ad90 con 0x7f4bdc1430c0 2024-03-25T18:16:43.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.471+0000 7f4be1cb1640 10 monclient: get_auth_request con 0x7f4bc006de80 auth_method 0 2024-03-25T18:16:43.473 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.471+0000 7f4bd27fc640 10 client.4547.objecter ms_handle_connect 0x7f4bc006de80 2024-03-25T18:16:43.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.477+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f4bd4054b40 con 0x7f4bdc1430c0 2024-03-25T18:16:43.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.477+0000 7f4bd27fc640 10 client.4547.objecter ms_dispatch 0x7f4bdc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:43.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.477+0000 7f4bd27fc640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:43.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.477+0000 7f4bd27fc640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:43.479 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.477+0000 7f4bd27fc640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-25T18:16:43.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.477+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4bd4050480 con 0x7f4bdc1430c0 2024-03-25T18:16:43.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.477+0000 7f4bd27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:43.480 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.477+0000 7f4bd27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:43.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.745+0000 7f4be3f15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:43.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.745+0000 7f4be3f15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:43.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.745+0000 7f4be3f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:16:43.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.745+0000 7f4be3f15640 1 -- 172.21.15.64:0/3375791893 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bdc0627b0 con 0x7f4bdc1430c0 2024-03-25T18:16:43.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.772+0000 7f4bd27fc640 1 -- 172.21.15.64:0/3375791893 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f4bd4052580 con 0x7f4bdc1430c0 2024-03-25T18:16:43.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.772+0000 7f4bd27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:43.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.772+0000 7f4bd27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:43.774 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:16:43.774 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:43.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.774+0000 7f4be3f15640 10 librados: watch_flush enter 2024-03-25T18:16:43.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.774+0000 7f4be3f15640 10 librados: watch_flush exit 2024-03-25T18:16:43.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.774+0000 7f4be3f15640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-25T18:16:43.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.774+0000 7f4be3f15640 10 client.4547.objecter successfully canceled tick 2024-03-25T18:16:43.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.774+0000 7f4be3f15640 1 -- 172.21.15.64:0/3375791893 >> v1:172.21.15.64:6800/2915543638 conn(0x7f4bc006de80 legacy=0x7f4bc0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.774+0000 7f4be3f15640 10 monclient: shutdown 2024-03-25T18:16:43.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.774+0000 7f4be3f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:43.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.774+0000 7f4be3f15640 1 -- 172.21.15.64:0/3375791893 >> v1:172.21.15.64:6790/0 conn(0x7f4bdc1430c0 legacy=0x7f4bdc1e2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.774+0000 7f4be3f15640 1 -- 172.21.15.64:0/3375791893 shutdown_connections 2024-03-25T18:16:43.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.774+0000 7f4be3f15640 1 -- 172.21.15.64:0/3375791893 >> 172.21.15.64:0/3375791893 conn(0x7f4bdc0b7ea0 msgr2=0x7f4bdc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:43.777 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.774+0000 7f4be3f15640 1 -- 172.21.15.64:0/3375791893 shutdown_connections 2024-03-25T18:16:43.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.774+0000 7f4be3f15640 1 -- 172.21.15.64:0/3375791893 wait complete. 2024-03-25T18:16:43.778 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:43.774+0000 7f4be3f15640 1 librados: shutdown 2024-03-25T18:16:43.787 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:50.594 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.710+0000 7f199c713640 10 monclient: get_monmap_and_config 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.710+0000 7f199c713640 10 monclient: build_initial_monmap 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.710+0000 7f199c713640 10 monclient: monmap: 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:16:50.712049+0000 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:16:50.712049+0000 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:50.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.711+0000 7f199c713640 1 Processor -- start 2024-03-25T18:16:50.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.711+0000 7f199c713640 1 -- start start 2024-03-25T18:16:50.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.711+0000 7f199c713640 10 monclient: init 2024-03-25T18:16:50.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.711+0000 7f199c713640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:50.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.711+0000 7f199c713640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:50.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.712+0000 7f199c713640 10 monclient(hunting): picked mon.noname-a con 0x7f19940a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:16:50.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.712+0000 7f199c713640 10 monclient(hunting): picked mon.noname-b con 0x7f199413f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:16:50.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.712+0000 7f199c713640 10 monclient(hunting): picked mon.noname-c con 0x7f19941430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:16:50.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.712+0000 7f199c713640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1994142520 con 0x7f19940a98f0 2024-03-25T18:16:50.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.712+0000 7f199c713640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19940b28a0 con 0x7f19941430c0 2024-03-25T18:16:50.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.712+0000 7f199c713640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19940b2a30 con 0x7f199413f710 2024-03-25T18:16:50.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.712+0000 7f199c713640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:50.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.712+0000 7f199c713640 10 monclient(hunting): authenticate will time out at 7396.951544s 2024-03-25T18:16:50.716 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.714+0000 7f199a4af640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f19940a98f0 0x7f19940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:36332/0 (socket says 172.21.15.64:36332) 2024-03-25T18:16:50.716 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.714+0000 7f199a4af640 1 -- 172.21.15.64:0/1934526172 learned_addr learned my addr 172.21.15.64:0/1934526172 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:16:50.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.722+0000 7f19994ad640 1 -- 172.21.15.64:0/1934526172 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822786991 0 0) 0x7f1994142520 con 0x7f19940a98f0 2024-03-25T18:16:50.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.722+0000 7f19994ad640 10 monclient(hunting): _negotiate 2024-03-25T18:16:50.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.722+0000 7f19994ad640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:50.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.722+0000 7f19994ad640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:50.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.722+0000 7f19994ad640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:16:50.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.722+0000 7f19994ad640 1 -- 172.21.15.64:0/1934526172 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1980003520 con 0x7f19940a98f0 2024-03-25T18:16:50.734 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.732+0000 7f19994ad640 1 -- 172.21.15.64:0/1934526172 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3143308647 0 0) 0x7f1980003520 con 0x7f19940a98f0 2024-03-25T18:16:50.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.732+0000 7f19994ad640 1 -- 172.21.15.64:0/1934526172 >> v1:172.21.15.64:6790/0 conn(0x7f199413f710 legacy=0x7f19941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:50.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.732+0000 7f19994ad640 1 -- 172.21.15.64:0/1934526172 >> v1:172.21.15.86:6789/0 conn(0x7f19941430c0 legacy=0x7f199414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:50.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.732+0000 7f19994ad640 10 monclient: _finish_hunting 0 2024-03-25T18:16:50.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.732+0000 7f19994ad640 1 monclient: found mon.noname-a 2024-03-25T18:16:50.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.732+0000 7f19994ad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:50.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.732+0000 7f19994ad640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:16:50.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.732+0000 7f19994ad640 1 -- 172.21.15.64:0/1934526172 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19940b44c0 con 0x7f19940a98f0 2024-03-25T18:16:50.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.732+0000 7f19994ad640 10 monclient: _finish_auth 0 2024-03-25T18:16:50.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.732+0000 7f19994ad640 10 monclient: _check_auth_tickets 2024-03-25T18:16:50.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.732+0000 7f19994ad640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:50.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.732+0000 7f19994ad640 1 -- 172.21.15.64:0/1934526172 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1990001690 con 0x7f19940a98f0 2024-03-25T18:16:50.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.732+0000 7f19994ad640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:50.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.732+0000 7f19994ad640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:50.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.732+0000 7f19994ad640 10 monclient: dump: 2024-03-25T18:16:50.735 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:50.736 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:50.736 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:50.736 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:50.736 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:50.736 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:50.736 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:50.736 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:50.736 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:50.736 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:50.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.732+0000 7f199c713640 5 monclient: authenticate success, global_id 4551 2024-03-25T18:16:50.736 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.732+0000 7f199c713640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:50.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.746+0000 7f19994ad640 1 -- 172.21.15.64:0/1934526172 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19900017f0 con 0x7f19940a98f0 2024-03-25T18:16:50.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.746+0000 7f19994ad640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:50.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.746+0000 7f19994ad640 1 -- 172.21.15.64:0/1934526172 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f1990001c40 con 0x7f19940a98f0 2024-03-25T18:16:50.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.746+0000 7f19994ad640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:50.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.746+0000 7f19994ad640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:50.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.746+0000 7f19994ad640 10 monclient: dump: 2024-03-25T18:16:50.748 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:50.748 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:50.748 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:50.748 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:50.748 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:50.749 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:50.749 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:50.749 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:50.749 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:50.749 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:50.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.746+0000 7f199c713640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:50.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.746+0000 7f199c713640 4 set_mon_vals no callback set 2024-03-25T18:16:50.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.747+0000 7f199c713640 10 monclient: shutdown 2024-03-25T18:16:50.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.747+0000 7f199c713640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:50.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.747+0000 7f199c713640 1 -- 172.21.15.64:0/1934526172 >> v1:172.21.15.64:6789/0 conn(0x7f19940a98f0 legacy=0x7f19940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:50.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.747+0000 7f199c713640 1 -- 172.21.15.64:0/1934526172 shutdown_connections 2024-03-25T18:16:50.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.747+0000 7f199c713640 1 -- 172.21.15.64:0/1934526172 >> 172.21.15.64:0/1934526172 conn(0x7f19940b7ea0 msgr2=0x7f19940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:50.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.747+0000 7f199c713640 1 -- 172.21.15.64:0/1934526172 shutdown_connections 2024-03-25T18:16:50.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.747+0000 7f199c713640 1 -- 172.21.15.64:0/1934526172 wait complete. 2024-03-25T18:16:50.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.748+0000 7f199c713640 10 monclient: build_initial_monmap 2024-03-25T18:16:50.751 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.748+0000 7f199c713640 10 monclient: monmap: 2024-03-25T18:16:50.751 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:16:50.751 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:50.751 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:16:50.751 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:16:50.751 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:50.751 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:16:50.752 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:16:50.752 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:16:50.752 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:16:50.752 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:50.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.748+0000 7f199c713640 1 librados: starting msgr at 2024-03-25T18:16:50.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.748+0000 7f199c713640 1 librados: starting objecter 2024-03-25T18:16:50.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.748+0000 7f199c713640 1 Processor -- start 2024-03-25T18:16:50.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.748+0000 7f199c713640 1 -- start start 2024-03-25T18:16:50.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.748+0000 7f199c713640 1 librados: setting wanted keys 2024-03-25T18:16:50.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.748+0000 7f199c713640 1 librados: calling monclient init 2024-03-25T18:16:50.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.748+0000 7f199c713640 10 monclient: init 2024-03-25T18:16:50.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.748+0000 7f199c713640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:50.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.748+0000 7f199c713640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:50.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.748+0000 7f199c713640 10 monclient(hunting): picked mon.noname-c con 0x7f19940a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:16:50.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.749+0000 7f199c713640 10 monclient(hunting): picked mon.noname-b con 0x7f199413f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:16:50.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.749+0000 7f199c713640 10 monclient(hunting): picked mon.noname-a con 0x7f19941430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:16:50.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.749+0000 7f199c713640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1994135ac0 con 0x7f19941430c0 2024-03-25T18:16:50.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.749+0000 7f199c713640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1994137030 con 0x7f199413f710 2024-03-25T18:16:50.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.749+0000 7f199c713640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19941372a0 con 0x7f19940a98f0 2024-03-25T18:16:50.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.749+0000 7f199c713640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:50.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.749+0000 7f199c713640 10 monclient(hunting): authenticate will time out at 7396.988507s 2024-03-25T18:16:50.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.762+0000 7f199acb0640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f19941430c0 0x7f19941e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:36358/0 (socket says 172.21.15.64:36358) 2024-03-25T18:16:50.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.762+0000 7f199acb0640 1 -- 172.21.15.64:0/4246807030 learned_addr learned my addr 172.21.15.64:0/4246807030 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:16:50.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.763+0000 7f198affd640 10 client.?.objecter ms_handle_connect 0x7f19941430c0 2024-03-25T18:16:50.765 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.763+0000 7f198affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:50.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.764+0000 7f198affd640 10 client.?.objecter ms_handle_connect 0x7f199413f710 2024-03-25T18:16:50.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.764+0000 7f198affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:50.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.769+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055508160 0 0) 0x7f1994135ac0 con 0x7f19941430c0 2024-03-25T18:16:50.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.769+0000 7f198affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:50.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.769+0000 7f198affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:50.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.769+0000 7f198affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:50.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.769+0000 7f198affd640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:16:50.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.769+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1978004eb0 con 0x7f19941430c0 2024-03-25T18:16:50.776 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.775+0000 7f198affd640 10 client.?.objecter ms_handle_connect 0x7f19940a98f0 2024-03-25T18:16:50.776 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.775+0000 7f198affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:50.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.778+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700897729 0 0) 0x7f1994137030 con 0x7f199413f710 2024-03-25T18:16:50.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.778+0000 7f198affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:50.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.778+0000 7f198affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:50.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.778+0000 7f198affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:50.780 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.778+0000 7f198affd640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:16:50.781 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.778+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1994135ac0 con 0x7f199413f710 2024-03-25T18:16:50.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.786+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449892123 0 0) 0x7f19941372a0 con 0x7f19940a98f0 2024-03-25T18:16:50.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.786+0000 7f198affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:50.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.786+0000 7f198affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:50.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.786+0000 7f198affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:50.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.786+0000 7f198affd640 10 monclient(hunting): my global_id is 4553 2024-03-25T18:16:50.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.786+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1994137030 con 0x7f19940a98f0 2024-03-25T18:16:50.795 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.793+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2565757199 0 0) 0x7f1978004eb0 con 0x7f19941430c0 2024-03-25T18:16:50.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.793+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19941372a0 con 0x7f19941430c0 2024-03-25T18:16:50.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.793+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f198c001c90 con 0x7f19941430c0 2024-03-25T18:16:50.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.793+0000 7f198affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:50.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.793+0000 7f198affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:50.796 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.793+0000 7f198affd640 10 monclient(hunting): dump: 2024-03-25T18:16:50.796 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:50.796 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:50.796 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:50.796 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:50.796 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:50.796 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:50.796 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:50.796 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:50.796 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:50.796 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:50.797 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.793+0000 7f198affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:50.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.800+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2338181300 0 0) 0x7f1994135ac0 con 0x7f199413f710 2024-03-25T18:16:50.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.800+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1978004eb0 con 0x7f199413f710 2024-03-25T18:16:50.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.800+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f19840018a0 con 0x7f199413f710 2024-03-25T18:16:50.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.800+0000 7f198affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:50.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.800+0000 7f198affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:50.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.800+0000 7f198affd640 10 monclient(hunting): dump: 2024-03-25T18:16:50.802 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:50.802 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:50.803 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:50.803 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:50.803 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:50.803 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:50.803 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:50.803 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:50.803 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:50.803 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:50.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4088810060 0 0) 0x7f19941372a0 con 0x7f19941430c0 2024-03-25T18:16:50.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 >> v1:172.21.15.64:6790/0 conn(0x7f19940a98f0 legacy=0x7f1994133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:50.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 >> v1:172.21.15.86:6789/0 conn(0x7f199413f710 legacy=0x7f19941379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:50.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f198affd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:50.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f198affd640 1 monclient: found mon.a 2024-03-25T18:16:50.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f198affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:50.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f198affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:50.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1994137580 con 0x7f19941430c0 2024-03-25T18:16:50.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f198affd640 10 monclient: _finish_auth 0 2024-03-25T18:16:50.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f198affd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:50.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f198affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:50.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f199c713640 5 monclient: authenticate success, global_id 4554 2024-03-25T18:16:50.805 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f199c713640 10 monclient: _renew_subs 2024-03-25T18:16:50.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f199c713640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:50.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f199c713640 1 -- 172.21.15.64:0/4246807030 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19941363b0 con 0x7f19941430c0 2024-03-25T18:16:50.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f199c713640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:50.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f199c713640 10 monclient: _renew_subs 2024-03-25T18:16:50.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f199c713640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:50.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f199c713640 1 -- 172.21.15.64:0/4246807030 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19941353a0 con 0x7f19941430c0 2024-03-25T18:16:50.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.802+0000 7f199c713640 1 librados: init done 2024-03-25T18:16:50.806 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.803+0000 7f199c713640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:50.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.804+0000 7f199c713640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:50.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.804+0000 7f199c713640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:50.807 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:50.804+0000 7f199c713640 1 -- 172.21.15.64:0/4246807030 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1994130d90 con 0x7f19941430c0 2024-03-25T18:16:51.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.123+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f198c003990 con 0x7f19941430c0 2024-03-25T18:16:51.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.123+0000 7f198affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:51.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.123+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f198c003bc0 con 0x7f19941430c0 2024-03-25T18:16:51.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.123+0000 7f198affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:51.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.123+0000 7f198affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:51.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.123+0000 7f198affd640 10 monclient: dump: 2024-03-25T18:16:51.125 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:51.125 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:51.125 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:51.125 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:51.125 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:51.125 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:51.125 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:51.125 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:51.125 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:51.125 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:51.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.123+0000 7f1998cac640 4 set_mon_vals no callback set 2024-03-25T18:16:51.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.124+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f198c003de0 con 0x7f19941430c0 2024-03-25T18:16:51.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.125+0000 7f199a4af640 10 monclient: get_auth_request con 0x7f197806df70 auth_method 0 2024-03-25T18:16:51.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.125+0000 7f198affd640 10 client.4554.objecter ms_handle_connect 0x7f197806df70 2024-03-25T18:16:51.132 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.131+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f198c0047e0 con 0x7f19941430c0 2024-03-25T18:16:51.132 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.131+0000 7f198affd640 10 client.4554.objecter ms_dispatch 0x7f1994000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:51.132 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.131+0000 7f198affd640 3 client.4554.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:51.133 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.131+0000 7f198affd640 3 client.4554.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:51.133 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.131+0000 7f198affd640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-25T18:16:51.133 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.131+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f198c0504d0 con 0x7f19941430c0 2024-03-25T18:16:51.133 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.131+0000 7f198affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:51.133 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.131+0000 7f198affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:51.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.385+0000 7f199c713640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:51.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.385+0000 7f199c713640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:51.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.385+0000 7f199c713640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:51.386 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.385+0000 7f199c713640 1 -- 172.21.15.64:0/4246807030 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1994074cc0 con 0x7f19941430c0 2024-03-25T18:16:51.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.413+0000 7f198affd640 1 -- 172.21.15.64:0/4246807030 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f198c052400 con 0x7f19941430c0 2024-03-25T18:16:51.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.414+0000 7f198affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:51.415 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.414+0000 7f198affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:51.415 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:16:51.415 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:51.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.416+0000 7f199c713640 10 librados: watch_flush enter 2024-03-25T18:16:51.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.416+0000 7f199c713640 10 librados: watch_flush exit 2024-03-25T18:16:51.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.416+0000 7f199c713640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-25T18:16:51.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.416+0000 7f199c713640 10 client.4554.objecter successfully canceled tick 2024-03-25T18:16:51.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.416+0000 7f199c713640 1 -- 172.21.15.64:0/4246807030 >> v1:172.21.15.64:6800/2915543638 conn(0x7f197806df70 legacy=0x7f1978070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:51.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.416+0000 7f199c713640 10 monclient: shutdown 2024-03-25T18:16:51.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.416+0000 7f199c713640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:51.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.416+0000 7f199c713640 1 -- 172.21.15.64:0/4246807030 >> v1:172.21.15.64:6789/0 conn(0x7f19941430c0 legacy=0x7f19941e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:51.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.416+0000 7f199c713640 1 -- 172.21.15.64:0/4246807030 shutdown_connections 2024-03-25T18:16:51.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.416+0000 7f199c713640 1 -- 172.21.15.64:0/4246807030 >> 172.21.15.64:0/4246807030 conn(0x7f19940b7ea0 msgr2=0x7f1994074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:51.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.416+0000 7f199c713640 1 -- 172.21.15.64:0/4246807030 shutdown_connections 2024-03-25T18:16:51.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.416+0000 7f199c713640 1 -- 172.21.15.64:0/4246807030 wait complete. 2024-03-25T18:16:51.418 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:51.416+0000 7f199c713640 1 librados: shutdown 2024-03-25T18:16:51.428 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:57.719 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:07:40.832 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:57.777 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:07:40.890 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:57.778 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:07:40.891 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:57.784 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:07:40.896 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:57.822 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:07:40.935 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:57.829 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:07:40.942 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:57.854 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:07:40.933 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:57.860 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:07:40.939 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:57.885 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:07:40.964 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:58.235 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:58.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.347+0000 7f281da82640 10 monclient: get_monmap_and_config 2024-03-25T18:16:58.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.347+0000 7f281da82640 10 monclient: build_initial_monmap 2024-03-25T18:16:58.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.347+0000 7f281da82640 10 monclient: monmap: 2024-03-25T18:16:58.348 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:16:58.348 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:58.349 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:16:58.348469+0000 2024-03-25T18:16:58.349 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:16:58.348469+0000 2024-03-25T18:16:58.349 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:58.349 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:58.349 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:16:58.349 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:16:58.349 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:16:58.349 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:58.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.348+0000 7f281da82640 1 Processor -- start 2024-03-25T18:16:58.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.348+0000 7f281da82640 1 -- start start 2024-03-25T18:16:58.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.348+0000 7f281da82640 10 monclient: init 2024-03-25T18:16:58.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.348+0000 7f281da82640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:58.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.348+0000 7f281da82640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:58.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.349+0000 7f281da82640 10 monclient(hunting): picked mon.noname-c con 0x7f28180b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:16:58.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.349+0000 7f281da82640 10 monclient(hunting): picked mon.noname-a con 0x7f28180b1730 addr v1:172.21.15.64:6789/0 2024-03-25T18:16:58.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.349+0000 7f281da82640 10 monclient(hunting): picked mon.noname-b con 0x7f281814be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:16:58.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.349+0000 7f281da82640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28180af940 con 0x7f28180b1730 2024-03-25T18:16:58.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.349+0000 7f281da82640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28180b0a70 con 0x7f28180b4d60 2024-03-25T18:16:58.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.349+0000 7f281da82640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28180b0d50 con 0x7f281814be90 2024-03-25T18:16:58.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.349+0000 7f281da82640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:58.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.349+0000 7f281da82640 10 monclient(hunting): authenticate will time out at 7404.587557s 2024-03-25T18:16:58.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.350+0000 7f2817fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f281814be90 0x7f281814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:58064/0 (socket says 172.21.15.64:58064) 2024-03-25T18:16:58.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.350+0000 7f2817fff640 1 -- 172.21.15.64:0/2680620405 learned_addr learned my addr 172.21.15.64:0/2680620405 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:16:58.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.362+0000 7f28167fc640 1 -- 172.21.15.64:0/2680620405 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058804143 0 0) 0x7f28180b0d50 con 0x7f281814be90 2024-03-25T18:16:58.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.362+0000 7f28167fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:58.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.362+0000 7f28167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:58.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.362+0000 7f28167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:58.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.362+0000 7f28167fc640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:16:58.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.362+0000 7f28167fc640 1 -- 172.21.15.64:0/2680620405 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2800003520 con 0x7f281814be90 2024-03-25T18:16:58.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.368+0000 7f28167fc640 1 -- 172.21.15.64:0/2680620405 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4160544095 0 0) 0x7f2800003520 con 0x7f281814be90 2024-03-25T18:16:58.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.368+0000 7f28167fc640 1 -- 172.21.15.64:0/2680620405 >> v1:172.21.15.86:6789/0 conn(0x7f28180b4d60 legacy=0x7f281813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:58.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.368+0000 7f28167fc640 1 -- 172.21.15.64:0/2680620405 >> v1:172.21.15.64:6789/0 conn(0x7f28180b1730 legacy=0x7f28180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:58.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.368+0000 7f28167fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:58.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.368+0000 7f28167fc640 1 monclient: found mon.noname-b 2024-03-25T18:16:58.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.368+0000 7f28167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:58.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.368+0000 7f28167fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:16:58.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.368+0000 7f28167fc640 1 -- 172.21.15.64:0/2680620405 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28180b1030 con 0x7f281814be90 2024-03-25T18:16:58.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.368+0000 7f28167fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:58.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.368+0000 7f28167fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:58.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.368+0000 7f28167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:58.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.368+0000 7f28167fc640 1 -- 172.21.15.64:0/2680620405 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2810001a10 con 0x7f281814be90 2024-03-25T18:16:58.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.368+0000 7f28167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:58.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.368+0000 7f28167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:58.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.368+0000 7f28167fc640 10 monclient: dump: 2024-03-25T18:16:58.370 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:58.371 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:58.371 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:58.371 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:58.371 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:58.371 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:58.371 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:58.371 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:58.371 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:58.371 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:58.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.368+0000 7f281da82640 5 monclient: authenticate success, global_id 4556 2024-03-25T18:16:58.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.368+0000 7f281da82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:58.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.375+0000 7f28167fc640 1 -- 172.21.15.64:0/2680620405 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2810001b70 con 0x7f281814be90 2024-03-25T18:16:58.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.375+0000 7f28167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:58.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.375+0000 7f28167fc640 1 -- 172.21.15.64:0/2680620405 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2810001f80 con 0x7f281814be90 2024-03-25T18:16:58.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.375+0000 7f281da82640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:58.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.375+0000 7f281da82640 4 set_mon_vals no callback set 2024-03-25T18:16:58.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.375+0000 7f28167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:58.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.375+0000 7f28167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:58.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.375+0000 7f28167fc640 10 monclient: dump: 2024-03-25T18:16:58.377 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:58.377 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:58.377 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:58.377 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:58.377 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:58.377 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:58.377 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:58.377 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:58.377 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:58.377 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.376+0000 7f281da82640 10 monclient: shutdown 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.376+0000 7f281da82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.376+0000 7f281da82640 1 -- 172.21.15.64:0/2680620405 >> v1:172.21.15.64:6790/0 conn(0x7f281814be90 legacy=0x7f281814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.376+0000 7f281da82640 1 -- 172.21.15.64:0/2680620405 shutdown_connections 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.376+0000 7f281da82640 1 -- 172.21.15.64:0/2680620405 >> 172.21.15.64:0/2680620405 conn(0x7f28180617a0 msgr2=0x7f2818061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.376+0000 7f281da82640 1 -- 172.21.15.64:0/2680620405 shutdown_connections 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.376+0000 7f281da82640 1 -- 172.21.15.64:0/2680620405 wait complete. 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.377+0000 7f281da82640 10 monclient: build_initial_monmap 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.377+0000 7f281da82640 10 monclient: monmap: 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:16:58.380 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:58.380 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:16:58.380 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:16:58.380 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:58.380 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:16:58.380 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:16:58.380 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:16:58.380 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:16:58.380 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:58.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.377+0000 7f281da82640 1 librados: starting msgr at 2024-03-25T18:16:58.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.377+0000 7f281da82640 1 librados: starting objecter 2024-03-25T18:16:58.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.377+0000 7f281da82640 1 Processor -- start 2024-03-25T18:16:58.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.378+0000 7f281da82640 1 -- start start 2024-03-25T18:16:58.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.378+0000 7f281da82640 1 librados: setting wanted keys 2024-03-25T18:16:58.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.378+0000 7f281da82640 1 librados: calling monclient init 2024-03-25T18:16:58.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.378+0000 7f281da82640 10 monclient: init 2024-03-25T18:16:58.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.378+0000 7f281da82640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:58.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.378+0000 7f281da82640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:58.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.378+0000 7f281da82640 10 monclient(hunting): picked mon.noname-c con 0x7f28180b1730 addr v1:172.21.15.64:6790/0 2024-03-25T18:16:58.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.379+0000 7f281da82640 10 monclient(hunting): picked mon.noname-b con 0x7f28180b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:16:58.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.379+0000 7f281da82640 10 monclient(hunting): picked mon.noname-a con 0x7f281814be90 addr v1:172.21.15.64:6789/0 2024-03-25T18:16:58.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.379+0000 7f281da82640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2818140760 con 0x7f281814be90 2024-03-25T18:16:58.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.379+0000 7f281da82640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2818140380 con 0x7f28180b4d60 2024-03-25T18:16:58.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.379+0000 7f281da82640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2818140510 con 0x7f28180b1730 2024-03-25T18:16:58.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.379+0000 7f281da82640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:58.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.379+0000 7f281da82640 10 monclient(hunting): authenticate will time out at 7404.617572s 2024-03-25T18:16:58.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.381+0000 7f2816ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f28180b4d60 0x7f28181db2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:55348/0 (socket says 172.21.15.64:55348) 2024-03-25T18:16:58.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.381+0000 7f2816ffd640 1 -- 172.21.15.64:0/2854721664 learned_addr learned my addr 172.21.15.64:0/2854721664 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:16:58.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.387+0000 7f27fffff640 10 client.?.objecter ms_handle_connect 0x7f28180b4d60 2024-03-25T18:16:58.388 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.387+0000 7f27fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:58.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.392+0000 7f2816ffd640 1 -- 172.21.15.64:0/2854721664 >> v1:172.21.15.86:6789/0 conn(0x7f28180b4d60 legacy=0x7f28181db2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:16:58.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.392+0000 7f2816ffd640 1 -- 172.21.15.64:0/2854721664 >> v1:172.21.15.86:6789/0 conn(0x7f28180b4d60 legacy=0x7f28181db2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:16:58.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.392+0000 7f2816ffd640 1 --1- 172.21.15.64:0/2854721664 >> v1:172.21.15.86:6789/0 conn(0x7f28180b4d60 0x7f28181db2a0 :-1 s=OPENED pgs=53222 cs=1 l=1).handle_message read tag failed 2024-03-25T18:16:58.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.392+0000 7f2816ffd640 1 --1- 172.21.15.64:0/2854721664 >> v1:172.21.15.86:6789/0 conn(0x7f28180b4d60 0x7f28181db2a0 :-1 s=OPENED pgs=53222 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:16:58.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.392+0000 7f27fffff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.86:6789/0 2024-03-25T18:16:58.397 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.397+0000 7f27fffff640 10 client.?.objecter ms_handle_connect 0x7f281814be90 2024-03-25T18:16:58.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.397+0000 7f27fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:58.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.398+0000 7f27fffff640 10 client.?.objecter ms_handle_connect 0x7f28180b1730 2024-03-25T18:16:58.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.398+0000 7f27fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:58.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.411+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182543684 0 0) 0x7f2818140760 con 0x7f281814be90 2024-03-25T18:16:58.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.411+0000 7f27fffff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:58.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.411+0000 7f27fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:58.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.411+0000 7f27fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:58.412 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.411+0000 7f27fffff640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:16:58.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.412+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2804004eb0 con 0x7f281814be90 2024-03-25T18:16:58.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.412+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348212911 0 0) 0x7f2818140510 con 0x7f28180b1730 2024-03-25T18:16:58.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.412+0000 7f27fffff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:58.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.412+0000 7f27fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:58.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.412+0000 7f27fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:58.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.412+0000 7f27fffff640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:16:58.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.412+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2818140760 con 0x7f28180b1730 2024-03-25T18:16:58.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.429+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3055637073 0 0) 0x7f2804004eb0 con 0x7f281814be90 2024-03-25T18:16:58.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.429+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2818140510 con 0x7f281814be90 2024-03-25T18:16:58.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.429+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2810001e80 con 0x7f281814be90 2024-03-25T18:16:58.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.429+0000 7f27fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:58.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.429+0000 7f27fffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:58.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.429+0000 7f27fffff640 10 monclient(hunting): dump: 2024-03-25T18:16:58.431 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:58.431 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:58.431 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:58.431 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:58.431 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:58.431 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:58.431 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:58.431 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:58.431 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:58.431 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:58.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.429+0000 7f27fffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:58.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.434+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4287022752 0 0) 0x7f2818140760 con 0x7f28180b1730 2024-03-25T18:16:58.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.434+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2804004eb0 con 0x7f28180b1730 2024-03-25T18:16:58.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.434+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2808001870 con 0x7f28180b1730 2024-03-25T18:16:58.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.434+0000 7f27fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:58.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.434+0000 7f27fffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:58.436 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.434+0000 7f27fffff640 10 monclient(hunting): dump: 2024-03-25T18:16:58.436 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:58.436 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:58.436 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:58.436 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:58.436 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:58.436 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:58.436 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:58.436 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:58.436 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:58.436 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:58.445 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 785889630 0 0) 0x7f2818140510 con 0x7f281814be90 2024-03-25T18:16:58.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 >> v1:172.21.15.64:6790/0 conn(0x7f28180b1730 legacy=0x7f28181dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:58.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 >> v1:172.21.15.86:6789/0 conn(0x7f28180b4d60 legacy=0x7f28181db2a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:16:58.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f27fffff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:58.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f27fffff640 1 monclient: found mon.a 2024-03-25T18:16:58.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f27fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:58.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f27fffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:58.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2818137530 con 0x7f281814be90 2024-03-25T18:16:58.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f27fffff640 10 monclient: _finish_auth 0 2024-03-25T18:16:58.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f27fffff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:58.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f27fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:58.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f281da82640 5 monclient: authenticate success, global_id 4560 2024-03-25T18:16:58.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f281da82640 10 monclient: _renew_subs 2024-03-25T18:16:58.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f281da82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:58.446 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f281da82640 1 -- 172.21.15.64:0/2854721664 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2818138070 con 0x7f281814be90 2024-03-25T18:16:58.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f281da82640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:58.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f281da82640 10 monclient: _renew_subs 2024-03-25T18:16:58.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f281da82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:58.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f281da82640 1 -- 172.21.15.64:0/2854721664 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2818145000 con 0x7f281814be90 2024-03-25T18:16:58.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.444+0000 7f281da82640 1 librados: init done 2024-03-25T18:16:58.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.445+0000 7f281da82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:58.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.445+0000 7f281da82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:58.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.445+0000 7f281da82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:58.447 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.445+0000 7f281da82640 1 -- 172.21.15.64:0/2854721664 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28181408f0 con 0x7f281814be90 2024-03-25T18:16:58.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.655+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2810002020 con 0x7f281814be90 2024-03-25T18:16:58.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.655+0000 7f27fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:58.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.655+0000 7f2815ffb640 4 set_mon_vals no callback set 2024-03-25T18:16:58.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.765+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2810004fb0 con 0x7f281814be90 2024-03-25T18:16:58.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.765+0000 7f27fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:58.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.765+0000 7f27fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:58.766 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.765+0000 7f27fffff640 10 monclient: dump: 2024-03-25T18:16:58.766 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:16:58.766 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:16:58.766 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:58.766 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:16:58.766 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:58.767 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:16:58.767 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:16:58.767 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:16:58.767 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:16:58.767 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:16:58.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.766+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f281001af80 con 0x7f281814be90 2024-03-25T18:16:58.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.766+0000 7f28177fe640 10 monclient: get_auth_request con 0x7f280406de90 auth_method 0 2024-03-25T18:16:58.767 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.766+0000 7f27fffff640 10 client.4560.objecter ms_handle_connect 0x7f280406de90 2024-03-25T18:16:58.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.773+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f2810054990 con 0x7f281814be90 2024-03-25T18:16:58.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.773+0000 7f27fffff640 10 client.4560.objecter ms_dispatch 0x7f2818000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:58.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.773+0000 7f27fffff640 3 client.4560.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:58.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.773+0000 7f27fffff640 3 client.4560.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:58.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.773+0000 7f27fffff640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-25T18:16:58.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.773+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f28100504a0 con 0x7f281814be90 2024-03-25T18:16:58.774 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.773+0000 7f27fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:58.775 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:58.773+0000 7f27fffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:59.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.027+0000 7f281da82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:59.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.027+0000 7f281da82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:59.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.027+0000 7f281da82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:16:59.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.027+0000 7f281da82640 1 -- 172.21.15.64:0/2854721664 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28180627b0 con 0x7f281814be90 2024-03-25T18:16:59.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.092+0000 7f27fffff640 1 -- 172.21.15.64:0/2854721664 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f28100523d0 con 0x7f281814be90 2024-03-25T18:16:59.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.092+0000 7f27fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:59.093 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.092+0000 7f27fffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:59.093 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:16:59.093 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:59.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.094+0000 7f281da82640 10 librados: watch_flush enter 2024-03-25T18:16:59.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.094+0000 7f281da82640 10 librados: watch_flush exit 2024-03-25T18:16:59.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.094+0000 7f281da82640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-25T18:16:59.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.094+0000 7f281da82640 10 client.4560.objecter successfully canceled tick 2024-03-25T18:16:59.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.095+0000 7f281da82640 1 -- 172.21.15.64:0/2854721664 >> v1:172.21.15.64:6800/2915543638 conn(0x7f280406de90 legacy=0x7f2804070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:59.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.095+0000 7f281da82640 10 monclient: shutdown 2024-03-25T18:16:59.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.095+0000 7f281da82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:59.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.095+0000 7f281da82640 1 -- 172.21.15.64:0/2854721664 >> v1:172.21.15.64:6789/0 conn(0x7f281814be90 legacy=0x7f28181e20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:59.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.095+0000 7f281da82640 1 -- 172.21.15.64:0/2854721664 shutdown_connections 2024-03-25T18:16:59.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.095+0000 7f281da82640 1 -- 172.21.15.64:0/2854721664 >> 172.21.15.64:0/2854721664 conn(0x7f28180617a0 msgr2=0x7f281813c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:59.096 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.095+0000 7f281da82640 1 -- 172.21.15.64:0/2854721664 shutdown_connections 2024-03-25T18:16:59.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.095+0000 7f281da82640 1 -- 172.21.15.64:0/2854721664 wait complete. 2024-03-25T18:16:59.097 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:16:59.095+0000 7f281da82640 1 librados: shutdown 2024-03-25T18:16:59.109 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:05.915 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:06.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.018+0000 7f0061600640 10 monclient: get_monmap_and_config 2024-03-25T18:17:06.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.018+0000 7f0061600640 10 monclient: build_initial_monmap 2024-03-25T18:17:06.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.018+0000 7f0061600640 10 monclient: monmap: 2024-03-25T18:17:06.019 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:17:06.019 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:06.019 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:17:06.019562+0000 2024-03-25T18:17:06.019 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:17:06.019562+0000 2024-03-25T18:17:06.020 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:06.020 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:06.020 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:17:06.020 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:17:06.020 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:17:06.020 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:06.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.019+0000 7f0061600640 1 Processor -- start 2024-03-25T18:17:06.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.019+0000 7f0061600640 1 -- start start 2024-03-25T18:17:06.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.019+0000 7f0061600640 10 monclient: init 2024-03-25T18:17:06.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.020+0000 7f0061600640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:06.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.020+0000 7f0061600640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:06.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.020+0000 7f0061600640 10 monclient(hunting): picked mon.noname-b con 0x7f005c0b4d60 addr v1:172.21.15.64:6790/0 2024-03-25T18:17:06.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.020+0000 7f0061600640 10 monclient(hunting): picked mon.noname-c con 0x7f005c0b1730 addr v1:172.21.15.86:6789/0 2024-03-25T18:17:06.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.020+0000 7f0061600640 10 monclient(hunting): picked mon.noname-a con 0x7f005c14be90 addr v1:172.21.15.64:6789/0 2024-03-25T18:17:06.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.020+0000 7f0061600640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005c0af940 con 0x7f005c14be90 2024-03-25T18:17:06.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.020+0000 7f0061600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005c0b0a70 con 0x7f005c0b1730 2024-03-25T18:17:06.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.020+0000 7f0061600640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005c0b0d50 con 0x7f005c0b4d60 2024-03-25T18:17:06.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.020+0000 7f0061600640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:06.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.020+0000 7f0061600640 10 monclient(hunting): authenticate will time out at 7412.258854s 2024-03-25T18:17:06.031 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.031+0000 7f0053fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f005c0b1730 0x7f005c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:60060/0 (socket says 172.21.15.64:60060) 2024-03-25T18:17:06.032 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.031+0000 7f0053fff640 1 -- 172.21.15.64:0/1880844892 learned_addr learned my addr 172.21.15.64:0/1880844892 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:17:06.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.042+0000 7f005a7fc640 1 -- 172.21.15.64:0/1880844892 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792817146 0 0) 0x7f005c0b0d50 con 0x7f005c0b4d60 2024-03-25T18:17:06.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.042+0000 7f005a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.042+0000 7f005a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.042+0000 7f005a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.042+0000 7f005a7fc640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:17:06.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.042+0000 7f005a7fc640 1 -- 172.21.15.64:0/1880844892 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0048003520 con 0x7f005c0b4d60 2024-03-25T18:17:06.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.045+0000 7f005a7fc640 1 -- 172.21.15.64:0/1880844892 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119469875 0 0) 0x7f005c0af940 con 0x7f005c14be90 2024-03-25T18:17:06.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.045+0000 7f005a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.045+0000 7f005a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.045+0000 7f005a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.045+0000 7f005a7fc640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:17:06.046 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.045+0000 7f005a7fc640 1 -- 172.21.15.64:0/1880844892 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f005c0b0d50 con 0x7f005c14be90 2024-03-25T18:17:06.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.055+0000 7f005a7fc640 1 -- 172.21.15.64:0/1880844892 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652015460 0 0) 0x7f005c0b0a70 con 0x7f005c0b1730 2024-03-25T18:17:06.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.055+0000 7f005a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.055+0000 7f005a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.055+0000 7f005a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.055+0000 7f005a7fc640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:17:06.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.055+0000 7f005a7fc640 1 -- 172.21.15.64:0/1880844892 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f005c0af940 con 0x7f005c0b1730 2024-03-25T18:17:06.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.073+0000 7f005a7fc640 1 -- 172.21.15.64:0/1880844892 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 229912208 0 0) 0x7f0048003520 con 0x7f005c0b4d60 2024-03-25T18:17:06.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.073+0000 7f005a7fc640 1 -- 172.21.15.64:0/1880844892 >> v1:172.21.15.86:6789/0 conn(0x7f005c0b1730 legacy=0x7f005c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.073+0000 7f005a7fc640 1 -- 172.21.15.64:0/1880844892 >> v1:172.21.15.64:6789/0 conn(0x7f005c14be90 legacy=0x7f005c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.073+0000 7f005a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:06.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.073+0000 7f005a7fc640 1 monclient: found mon.noname-b 2024-03-25T18:17:06.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.073+0000 7f005a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:06.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.073+0000 7f005a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:17:06.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.073+0000 7f005a7fc640 1 -- 172.21.15.64:0/1880844892 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005c0b1030 con 0x7f005c0b4d60 2024-03-25T18:17:06.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.073+0000 7f005a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:06.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.073+0000 7f005a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:06.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.073+0000 7f005a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:06.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.073+0000 7f005a7fc640 1 -- 172.21.15.64:0/1880844892 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f004c001690 con 0x7f005c0b4d60 2024-03-25T18:17:06.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.073+0000 7f0061600640 5 monclient: authenticate success, global_id 4562 2024-03-25T18:17:06.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.073+0000 7f005a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:06.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.073+0000 7f005a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:06.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.073+0000 7f005a7fc640 10 monclient: dump: 2024-03-25T18:17:06.076 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:06.076 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:06.076 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:06.076 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:06.076 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:06.076 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:06.076 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:06.076 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:06.076 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:06.076 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:06.076 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.073+0000 7f0061600640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:06.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.082+0000 7f005a7fc640 1 -- 172.21.15.64:0/1880844892 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f004c0017f0 con 0x7f005c0b4d60 2024-03-25T18:17:06.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.082+0000 7f005a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:06.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.082+0000 7f005a7fc640 1 -- 172.21.15.64:0/1880844892 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f004c001c00 con 0x7f005c0b4d60 2024-03-25T18:17:06.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.082+0000 7f005a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:06.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.082+0000 7f005a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:06.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.082+0000 7f005a7fc640 10 monclient: dump: 2024-03-25T18:17:06.083 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:06.084 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:06.084 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:06.084 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:06.084 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:06.084 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:06.084 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:06.084 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:06.084 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:06.084 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:06.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.082+0000 7f0061600640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:06.084 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.082+0000 7f0061600640 4 set_mon_vals no callback set 2024-03-25T18:17:06.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.083+0000 7f0061600640 10 monclient: shutdown 2024-03-25T18:17:06.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.083+0000 7f0061600640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:06.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.083+0000 7f0061600640 1 -- 172.21.15.64:0/1880844892 >> v1:172.21.15.64:6790/0 conn(0x7f005c0b4d60 legacy=0x7f005c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.083+0000 7f0061600640 1 -- 172.21.15.64:0/1880844892 shutdown_connections 2024-03-25T18:17:06.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.083+0000 7f0061600640 1 -- 172.21.15.64:0/1880844892 >> 172.21.15.64:0/1880844892 conn(0x7f005c0617a0 msgr2=0x7f005c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:06.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.083+0000 7f0061600640 1 -- 172.21.15.64:0/1880844892 shutdown_connections 2024-03-25T18:17:06.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.083+0000 7f0061600640 1 -- 172.21.15.64:0/1880844892 wait complete. 2024-03-25T18:17:06.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.084+0000 7f0061600640 10 monclient: build_initial_monmap 2024-03-25T18:17:06.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.084+0000 7f0061600640 10 monclient: monmap: 2024-03-25T18:17:06.086 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:17:06.086 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:06.086 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:17:06.086 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:17:06.086 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:06.086 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:17:06.086 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:17:06.086 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:17:06.086 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:17:06.086 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:06.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.085+0000 7f0061600640 1 librados: starting msgr at 2024-03-25T18:17:06.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.085+0000 7f0061600640 1 librados: starting objecter 2024-03-25T18:17:06.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.085+0000 7f0061600640 1 Processor -- start 2024-03-25T18:17:06.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.085+0000 7f0061600640 1 -- start start 2024-03-25T18:17:06.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.085+0000 7f0061600640 1 librados: setting wanted keys 2024-03-25T18:17:06.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.085+0000 7f0061600640 1 librados: calling monclient init 2024-03-25T18:17:06.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.085+0000 7f0061600640 10 monclient: init 2024-03-25T18:17:06.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.086+0000 7f0061600640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:06.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.086+0000 7f0061600640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:06.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.086+0000 7f0061600640 10 monclient(hunting): picked mon.noname-b con 0x7f005c0b1730 addr v1:172.21.15.86:6789/0 2024-03-25T18:17:06.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.086+0000 7f0061600640 10 monclient(hunting): picked mon.noname-a con 0x7f005c0b4d60 addr v1:172.21.15.64:6789/0 2024-03-25T18:17:06.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.086+0000 7f0061600640 10 monclient(hunting): picked mon.noname-c con 0x7f005c14be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:17:06.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.086+0000 7f0061600640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005c138010 con 0x7f005c0b4d60 2024-03-25T18:17:06.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.086+0000 7f0061600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005c1382f0 con 0x7f005c0b1730 2024-03-25T18:17:06.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.086+0000 7f0061600640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005c136d90 con 0x7f005c14be90 2024-03-25T18:17:06.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.086+0000 7f0061600640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:06.088 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.086+0000 7f0061600640 10 monclient(hunting): authenticate will time out at 7412.324614s 2024-03-25T18:17:06.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.089+0000 7f005affd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f005c0b1730 0x7f005c135f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:60120/0 (socket says 172.21.15.64:60120) 2024-03-25T18:17:06.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.090+0000 7f005affd640 1 -- 172.21.15.64:0/134194655 learned_addr learned my addr 172.21.15.64:0/134194655 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:17:06.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.094+0000 7f00537fe640 10 client.?.objecter ms_handle_connect 0x7f005c0b1730 2024-03-25T18:17:06.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.094+0000 7f00537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:06.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.103+0000 7f00537fe640 10 client.?.objecter ms_handle_connect 0x7f005c0b4d60 2024-03-25T18:17:06.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.103+0000 7f00537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:06.111 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.110+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195960607 0 0) 0x7f005c1382f0 con 0x7f005c0b1730 2024-03-25T18:17:06.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.110+0000 7f00537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.110+0000 7f00537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.110+0000 7f00537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.110+0000 7f00537fe640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:17:06.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.111+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f003c004eb0 con 0x7f005c0b1730 2024-03-25T18:17:06.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.111+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136141445 0 0) 0x7f005c138010 con 0x7f005c0b4d60 2024-03-25T18:17:06.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.111+0000 7f00537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.111+0000 7f00537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.111+0000 7f00537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.111+0000 7f00537fe640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:17:06.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.111+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f005c1382f0 con 0x7f005c0b4d60 2024-03-25T18:17:06.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.121+0000 7f00537fe640 10 client.?.objecter ms_handle_connect 0x7f005c14be90 2024-03-25T18:17:06.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.121+0000 7f00537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:06.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.123+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3814447738 0 0) 0x7f005c1382f0 con 0x7f005c0b4d60 2024-03-25T18:17:06.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.123+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f005c138010 con 0x7f005c0b4d60 2024-03-25T18:17:06.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.123+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f00440018b0 con 0x7f005c0b4d60 2024-03-25T18:17:06.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.123+0000 7f00537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:06.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.123+0000 7f00537fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:06.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.123+0000 7f00537fe640 10 monclient(hunting): dump: 2024-03-25T18:17:06.124 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:06.124 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:06.124 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:06.124 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:06.125 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:06.125 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:06.125 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:06.125 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:06.125 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:06.125 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:06.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.123+0000 7f00537fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:06.131 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.130+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688674963 0 0) 0x7f005c136d90 con 0x7f005c14be90 2024-03-25T18:17:06.131 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.130+0000 7f00537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.131 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.130+0000 7f00537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.131 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.130+0000 7f00537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.131 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.130+0000 7f00537fe640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:17:06.131 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.130+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f005c1382f0 con 0x7f005c14be90 2024-03-25T18:17:06.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.136+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1310057980 0 0) 0x7f003c004eb0 con 0x7f005c0b1730 2024-03-25T18:17:06.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.136+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f005c136d90 con 0x7f005c0b1730 2024-03-25T18:17:06.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.136+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f004c001bc0 con 0x7f005c0b1730 2024-03-25T18:17:06.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.136+0000 7f00537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:06.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.136+0000 7f00537fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:06.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.136+0000 7f00537fe640 10 monclient(hunting): dump: 2024-03-25T18:17:06.137 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:06.137 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:06.137 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:06.137 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:06.138 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:06.138 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:06.138 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:06.138 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:06.138 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:06.138 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:06.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.139+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2827056834 0 0) 0x7f005c138010 con 0x7f005c0b4d60 2024-03-25T18:17:06.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.139+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 >> v1:172.21.15.64:6790/0 conn(0x7f005c14be90 legacy=0x7f005c1427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.139+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 >> v1:172.21.15.86:6789/0 conn(0x7f005c0b1730 legacy=0x7f005c135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.139+0000 7f00537fe640 10 monclient: _finish_hunting 0 2024-03-25T18:17:06.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.139+0000 7f00537fe640 1 monclient: found mon.a 2024-03-25T18:17:06.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.139+0000 7f00537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:06.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.139+0000 7f00537fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:17:06.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.139+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005c144010 con 0x7f005c0b4d60 2024-03-25T18:17:06.141 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.139+0000 7f00537fe640 10 monclient: _finish_auth 0 2024-03-25T18:17:06.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.139+0000 7f00537fe640 10 monclient: _check_auth_tickets 2024-03-25T18:17:06.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.139+0000 7f00537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:06.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.139+0000 7f0061600640 5 monclient: authenticate success, global_id 4566 2024-03-25T18:17:06.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.139+0000 7f0061600640 10 monclient: _renew_subs 2024-03-25T18:17:06.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.139+0000 7f0061600640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:17:06.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.139+0000 7f0061600640 1 -- 172.21.15.64:0/134194655 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f005c1372b0 con 0x7f005c0b4d60 2024-03-25T18:17:06.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.139+0000 7f0061600640 10 client.4566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:06.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.140+0000 7f0061600640 10 monclient: _renew_subs 2024-03-25T18:17:06.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.140+0000 7f0061600640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:17:06.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.140+0000 7f0061600640 1 -- 172.21.15.64:0/134194655 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f005c1369c0 con 0x7f005c0b4d60 2024-03-25T18:17:06.142 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.140+0000 7f0061600640 1 librados: init done 2024-03-25T18:17:06.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.141+0000 7f0061600640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:06.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.141+0000 7f0061600640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:06.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.141+0000 7f0061600640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:17:06.143 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.141+0000 7f0061600640 1 -- 172.21.15.64:0/134194655 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f005c136f20 con 0x7f005c0b4d60 2024-03-25T18:17:06.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.459+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0044001d00 con 0x7f005c0b4d60 2024-03-25T18:17:06.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.459+0000 7f00537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:06.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.459+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0044004a00 con 0x7f005c0b4d60 2024-03-25T18:17:06.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.459+0000 7f00537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:06.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.460+0000 7f00537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:06.461 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.460+0000 7f00537fe640 10 monclient: dump: 2024-03-25T18:17:06.461 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:06.461 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:06.461 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:06.461 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:06.461 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:06.461 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:06.461 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:06.462 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:06.462 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:06.462 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:06.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.459+0000 7f0059ffb640 4 set_mon_vals no callback set 2024-03-25T18:17:06.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.460+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f0044004c20 con 0x7f005c0b4d60 2024-03-25T18:17:06.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.461+0000 7f005affd640 10 monclient: get_auth_request con 0x7f003c06dfb0 auth_method 0 2024-03-25T18:17:06.462 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.461+0000 7f00537fe640 10 client.4566.objecter ms_handle_connect 0x7f003c06dfb0 2024-03-25T18:17:06.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.470+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f0044054780 con 0x7f005c0b4d60 2024-03-25T18:17:06.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.470+0000 7f00537fe640 10 client.4566.objecter ms_dispatch 0x7f005c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:06.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.470+0000 7f00537fe640 3 client.4566.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:06.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.470+0000 7f00537fe640 3 client.4566.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:06.471 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.470+0000 7f00537fe640 20 client.4566.objecter dump_active .. 0 homeless 2024-03-25T18:17:06.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.470+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00440500c0 con 0x7f005c0b4d60 2024-03-25T18:17:06.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.470+0000 7f00537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:06.472 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.470+0000 7f00537fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:06.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.732+0000 7f0061600640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:06.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.732+0000 7f0061600640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:06.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.732+0000 7f0061600640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:17:06.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.732+0000 7f0061600640 1 -- 172.21.15.64:0/134194655 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f005c137440 con 0x7f005c0b4d60 2024-03-25T18:17:06.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.759+0000 7f00537fe640 1 -- 172.21.15.64:0/134194655 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f00440521c0 con 0x7f005c0b4d60 2024-03-25T18:17:06.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.759+0000 7f00537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:06.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.759+0000 7f00537fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:06.761 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:17:06.761 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:06.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.761+0000 7f0061600640 10 librados: watch_flush enter 2024-03-25T18:17:06.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.761+0000 7f0061600640 10 librados: watch_flush exit 2024-03-25T18:17:06.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.761+0000 7f0061600640 20 client.4566.objecter shutdown clearing up homeless session... 2024-03-25T18:17:06.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.761+0000 7f0061600640 10 client.4566.objecter successfully canceled tick 2024-03-25T18:17:06.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.761+0000 7f0061600640 1 -- 172.21.15.64:0/134194655 >> v1:172.21.15.64:6800/2915543638 conn(0x7f003c06dfb0 legacy=0x7f003c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.761+0000 7f0061600640 10 monclient: shutdown 2024-03-25T18:17:06.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.761+0000 7f0061600640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:06.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.761+0000 7f0061600640 1 -- 172.21.15.64:0/134194655 >> v1:172.21.15.64:6789/0 conn(0x7f005c0b4d60 legacy=0x7f005c143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.763 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.761+0000 7f0061600640 1 -- 172.21.15.64:0/134194655 shutdown_connections 2024-03-25T18:17:06.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.761+0000 7f0061600640 1 -- 172.21.15.64:0/134194655 >> 172.21.15.64:0/134194655 conn(0x7f005c0617a0 msgr2=0x7f005c0b2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:06.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.761+0000 7f0061600640 1 -- 172.21.15.64:0/134194655 shutdown_connections 2024-03-25T18:17:06.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.762+0000 7f0061600640 1 -- 172.21.15.64:0/134194655 wait complete. 2024-03-25T18:17:06.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:06.762+0000 7f0061600640 1 librados: shutdown 2024-03-25T18:17:06.775 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:13.581 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:13.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.693+0000 7fb7ecdcd640 10 monclient: get_monmap_and_config 2024-03-25T18:17:13.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.693+0000 7fb7ecdcd640 10 monclient: build_initial_monmap 2024-03-25T18:17:13.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.693+0000 7fb7ecdcd640 10 monclient: monmap: 2024-03-25T18:17:13.694 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:17:13.694 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:13.694 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:17:13.694127+0000 2024-03-25T18:17:13.694 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:17:13.694127+0000 2024-03-25T18:17:13.694 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:13.694 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:13.695 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:17:13.695 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:17:13.695 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:17:13.695 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:13.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.694+0000 7fb7ecdcd640 1 Processor -- start 2024-03-25T18:17:13.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.694+0000 7fb7ecdcd640 1 -- start start 2024-03-25T18:17:13.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.694+0000 7fb7ecdcd640 10 monclient: init 2024-03-25T18:17:13.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.694+0000 7fb7ecdcd640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:13.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.694+0000 7fb7ecdcd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:13.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.694+0000 7fb7ecdcd640 10 monclient(hunting): picked mon.noname-c con 0x7fb7e80a96e0 addr v1:172.21.15.86:6789/0 2024-03-25T18:17:13.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.694+0000 7fb7ecdcd640 10 monclient(hunting): picked mon.noname-b con 0x7fb7e80b1b10 addr v1:172.21.15.64:6790/0 2024-03-25T18:17:13.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.695+0000 7fb7ecdcd640 10 monclient(hunting): picked mon.noname-a con 0x7fb7e8146060 addr v1:172.21.15.64:6789/0 2024-03-25T18:17:13.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.695+0000 7fb7ecdcd640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e8140040 con 0x7fb7e8146060 2024-03-25T18:17:13.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.695+0000 7fb7ecdcd640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e80b1fa0 con 0x7fb7e80a96e0 2024-03-25T18:17:13.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.695+0000 7fb7ecdcd640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e80b24c0 con 0x7fb7e80b1b10 2024-03-25T18:17:13.696 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.695+0000 7fb7ecdcd640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:13.697 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.695+0000 7fb7ecdcd640 10 monclient(hunting): authenticate will time out at 7419.933535s 2024-03-25T18:17:13.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.700+0000 7fb7dffff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fb7e80b1b10 0x7fb7e8145950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53370/0 (socket says 172.21.15.64:53370) 2024-03-25T18:17:13.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.700+0000 7fb7dffff640 1 -- 172.21.15.64:0/4104035111 learned_addr learned my addr 172.21.15.64:0/4104035111 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:17:13.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.709+0000 7fb7e5d9b640 1 -- 172.21.15.64:0/4104035111 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701881189 0 0) 0x7fb7e80b24c0 con 0x7fb7e80b1b10 2024-03-25T18:17:13.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.709+0000 7fb7e5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:17:13.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.709+0000 7fb7e5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:13.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.709+0000 7fb7e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:13.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.709+0000 7fb7e5d9b640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:17:13.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.709+0000 7fb7e5d9b640 1 -- 172.21.15.64:0/4104035111 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d4003520 con 0x7fb7e80b1b10 2024-03-25T18:17:13.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.716+0000 7fb7e5d9b640 1 -- 172.21.15.64:0/4104035111 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1381495705 0 0) 0x7fb7d4003520 con 0x7fb7e80b1b10 2024-03-25T18:17:13.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.716+0000 7fb7e5d9b640 1 -- 172.21.15.64:0/4104035111 >> v1:172.21.15.86:6789/0 conn(0x7fb7e80a96e0 legacy=0x7fb7e80b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:13.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.716+0000 7fb7e5d9b640 1 -- 172.21.15.64:0/4104035111 >> v1:172.21.15.64:6789/0 conn(0x7fb7e8146060 legacy=0x7fb7e814dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:17:13.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.716+0000 7fb7e5d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:17:13.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.716+0000 7fb7e5d9b640 1 monclient: found mon.noname-b 2024-03-25T18:17:13.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.716+0000 7fb7e5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:13.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.716+0000 7fb7e5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:17:13.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.716+0000 7fb7e5d9b640 1 -- 172.21.15.64:0/4104035111 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e80b15b0 con 0x7fb7e80b1b10 2024-03-25T18:17:13.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.716+0000 7fb7e5d9b640 10 monclient: _finish_auth 0 2024-03-25T18:17:13.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.716+0000 7fb7e5d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:17:13.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.716+0000 7fb7e5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:13.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.716+0000 7fb7e5d9b640 1 -- 172.21.15.64:0/4104035111 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb7d0001810 con 0x7fb7e80b1b10 2024-03-25T18:17:13.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.716+0000 7fb7ecdcd640 5 monclient: authenticate success, global_id 4568 2024-03-25T18:17:13.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.716+0000 7fb7ecdcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:13.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.716+0000 7fb7e5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:13.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.716+0000 7fb7e5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:13.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.716+0000 7fb7e5d9b640 10 monclient: dump: 2024-03-25T18:17:13.719 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:13.719 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:13.719 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:13.719 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:13.719 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:13.719 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:13.719 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:13.719 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:13.720 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:13.720 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:13.720 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.716+0000 7fb7ecdcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:13.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.722+0000 7fb7e5d9b640 1 -- 172.21.15.64:0/4104035111 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7d00019e0 con 0x7fb7e80b1b10 2024-03-25T18:17:13.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.722+0000 7fb7e5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:13.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.722+0000 7fb7e5d9b640 1 -- 172.21.15.64:0/4104035111 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb7d0003610 con 0x7fb7e80b1b10 2024-03-25T18:17:13.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.722+0000 7fb7e5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:13.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.722+0000 7fb7e5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:13.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.722+0000 7fb7e5d9b640 10 monclient: dump: 2024-03-25T18:17:13.723 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:13.723 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:13.723 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:13.723 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:13.723 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:13.724 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:13.724 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:13.724 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:13.724 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:13.724 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:13.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.722+0000 7fb7ecdcd640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:13.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.722+0000 7fb7ecdcd640 4 set_mon_vals no callback set 2024-03-25T18:17:13.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.723+0000 7fb7ecdcd640 10 monclient: shutdown 2024-03-25T18:17:13.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.723+0000 7fb7ecdcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:13.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.723+0000 7fb7ecdcd640 1 -- 172.21.15.64:0/4104035111 >> v1:172.21.15.64:6790/0 conn(0x7fb7e80b1b10 legacy=0x7fb7e8145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:13.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.723+0000 7fb7ecdcd640 1 -- 172.21.15.64:0/4104035111 shutdown_connections 2024-03-25T18:17:13.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.723+0000 7fb7ecdcd640 1 -- 172.21.15.64:0/4104035111 >> 172.21.15.64:0/4104035111 conn(0x7fb7e80b7ea0 msgr2=0x7fb7e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:13.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.723+0000 7fb7ecdcd640 1 -- 172.21.15.64:0/4104035111 shutdown_connections 2024-03-25T18:17:13.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.723+0000 7fb7ecdcd640 1 -- 172.21.15.64:0/4104035111 wait complete. 2024-03-25T18:17:13.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.724+0000 7fb7ecdcd640 10 monclient: build_initial_monmap 2024-03-25T18:17:13.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.724+0000 7fb7ecdcd640 10 monclient: monmap: 2024-03-25T18:17:13.727 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:17:13.727 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:13.727 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:17:13.727 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:17:13.727 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:13.727 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:17:13.727 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:17:13.727 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:17:13.727 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:17:13.727 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:13.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.724+0000 7fb7ecdcd640 1 librados: starting msgr at 2024-03-25T18:17:13.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.724+0000 7fb7ecdcd640 1 librados: starting objecter 2024-03-25T18:17:13.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.724+0000 7fb7ecdcd640 1 Processor -- start 2024-03-25T18:17:13.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.724+0000 7fb7ecdcd640 1 -- start start 2024-03-25T18:17:13.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.724+0000 7fb7ecdcd640 1 librados: setting wanted keys 2024-03-25T18:17:13.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.724+0000 7fb7ecdcd640 1 librados: calling monclient init 2024-03-25T18:17:13.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.724+0000 7fb7ecdcd640 10 monclient: init 2024-03-25T18:17:13.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.725+0000 7fb7ecdcd640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:13.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.725+0000 7fb7ecdcd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:13.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.725+0000 7fb7ecdcd640 10 monclient(hunting): picked mon.noname-a con 0x7fb7e80a96e0 addr v1:172.21.15.64:6789/0 2024-03-25T18:17:13.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.725+0000 7fb7ecdcd640 10 monclient(hunting): picked mon.noname-c con 0x7fb7e8146060 addr v1:172.21.15.64:6790/0 2024-03-25T18:17:13.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.725+0000 7fb7ecdcd640 10 monclient(hunting): picked mon.noname-b con 0x7fb7e813acd0 addr v1:172.21.15.86:6789/0 2024-03-25T18:17:13.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.725+0000 7fb7ecdcd640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e80ad9b0 con 0x7fb7e80a96e0 2024-03-25T18:17:13.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.725+0000 7fb7ecdcd640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e80ade40 con 0x7fb7e813acd0 2024-03-25T18:17:13.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.725+0000 7fb7ecdcd640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e80aebb0 con 0x7fb7e8146060 2024-03-25T18:17:13.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.725+0000 7fb7ecdcd640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:13.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.725+0000 7fb7ecdcd640 10 monclient(hunting): authenticate will time out at 7419.963761s 2024-03-25T18:17:13.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.729+0000 7fb7dffff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fb7e8146060 0x7fb7e80af2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53390/0 (socket says 172.21.15.64:53390) 2024-03-25T18:17:13.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.729+0000 7fb7dffff640 1 -- 172.21.15.64:0/171944699 learned_addr learned my addr 172.21.15.64:0/171944699 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:17:13.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.730+0000 7fb7deffd640 10 client.?.objecter ms_handle_connect 0x7fb7e8146060 2024-03-25T18:17:13.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.730+0000 7fb7deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:13.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.735+0000 7fb7deffd640 10 client.?.objecter ms_handle_connect 0x7fb7e80a96e0 2024-03-25T18:17:13.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.735+0000 7fb7deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:13.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.736+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903194986 0 0) 0x7fb7e80aebb0 con 0x7fb7e8146060 2024-03-25T18:17:13.737 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.736+0000 7fb7deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:13.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.736+0000 7fb7deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:13.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.736+0000 7fb7deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:13.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.736+0000 7fb7deffd640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:17:13.738 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.736+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c8004eb0 con 0x7fb7e8146060 2024-03-25T18:17:13.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.739+0000 7fb7deffd640 10 client.?.objecter ms_handle_connect 0x7fb7e813acd0 2024-03-25T18:17:13.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.739+0000 7fb7deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:13.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.745+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2047043323 0 0) 0x7fb7c8004eb0 con 0x7fb7e8146060 2024-03-25T18:17:13.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.745+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7e80aebb0 con 0x7fb7e8146060 2024-03-25T18:17:13.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.745+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb7d0001530 con 0x7fb7e8146060 2024-03-25T18:17:13.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.745+0000 7fb7deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:13.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.745+0000 7fb7deffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:13.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.745+0000 7fb7deffd640 10 monclient(hunting): dump: 2024-03-25T18:17:13.746 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:13.746 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:13.746 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:13.746 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:13.746 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:13.747 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:13.747 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:13.747 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:13.747 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:13.747 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:13.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.745+0000 7fb7deffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:13.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.748+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959189081 0 0) 0x7fb7e80ad9b0 con 0x7fb7e80a96e0 2024-03-25T18:17:13.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.748+0000 7fb7deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:13.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.748+0000 7fb7deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:13.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.748+0000 7fb7deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:13.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.748+0000 7fb7deffd640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:17:13.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.748+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c8004eb0 con 0x7fb7e80a96e0 2024-03-25T18:17:13.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.753+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736596942 0 0) 0x7fb7e80ade40 con 0x7fb7e813acd0 2024-03-25T18:17:13.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.753+0000 7fb7deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:13.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.753+0000 7fb7deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:13.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.753+0000 7fb7deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:13.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.753+0000 7fb7deffd640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:17:13.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.753+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7e80ad9b0 con 0x7fb7e813acd0 2024-03-25T18:17:13.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1976870800 0 0) 0x7fb7e80aebb0 con 0x7fb7e8146060 2024-03-25T18:17:13.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 >> v1:172.21.15.86:6789/0 conn(0x7fb7e813acd0 legacy=0x7fb7e80af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:13.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 >> v1:172.21.15.64:6789/0 conn(0x7fb7e80a96e0 legacy=0x7fb7e813a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:13.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7deffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:13.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7deffd640 1 monclient: found mon.c 2024-03-25T18:17:13.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:13.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7deffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:17:13.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e80b0790 con 0x7fb7e8146060 2024-03-25T18:17:13.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7deffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:13.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7deffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:13.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:13.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7ecdcd640 5 monclient: authenticate success, global_id 4571 2024-03-25T18:17:13.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7ecdcd640 10 monclient: _renew_subs 2024-03-25T18:17:13.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7ecdcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:17:13.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7ecdcd640 1 -- 172.21.15.64:0/171944699 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e80aca30 con 0x7fb7e8146060 2024-03-25T18:17:13.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7ecdcd640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:13.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7ecdcd640 10 monclient: _renew_subs 2024-03-25T18:17:13.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7ecdcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:17:13.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7ecdcd640 1 -- 172.21.15.64:0/171944699 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e80ad300 con 0x7fb7e8146060 2024-03-25T18:17:13.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.756+0000 7fb7ecdcd640 1 librados: init done 2024-03-25T18:17:13.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.757+0000 7fb7ecdcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:13.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.757+0000 7fb7ecdcd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:13.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.757+0000 7fb7ecdcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:17:13.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:13.757+0000 7fb7ecdcd640 1 -- 172.21.15.64:0/171944699 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e80a9ac0 con 0x7fb7e8146060 2024-03-25T18:17:14.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.072+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7d0001790 con 0x7fb7e8146060 2024-03-25T18:17:14.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.072+0000 7fb7deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:14.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.072+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb7d0003d20 con 0x7fb7e8146060 2024-03-25T18:17:14.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.072+0000 7fb7deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:14.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.072+0000 7fb7deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:14.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.072+0000 7fb7deffd640 10 monclient: dump: 2024-03-25T18:17:14.074 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:14.074 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:14.074 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:14.074 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:14.074 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:14.074 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:14.074 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:14.074 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:14.074 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:14.074 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:14.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.072+0000 7fb7e559a640 4 set_mon_vals no callback set 2024-03-25T18:17:14.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.073+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fb7d0003f40 con 0x7fb7e8146060 2024-03-25T18:17:14.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.073+0000 7fb7e659c640 10 monclient: get_auth_request con 0x7fb7c806ddf0 auth_method 0 2024-03-25T18:17:14.075 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.073+0000 7fb7deffd640 10 client.4571.objecter ms_handle_connect 0x7fb7c806ddf0 2024-03-25T18:17:14.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.081+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fb7d00548a0 con 0x7fb7e8146060 2024-03-25T18:17:14.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.081+0000 7fb7deffd640 10 client.4571.objecter ms_dispatch 0x7fb7e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:14.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.081+0000 7fb7deffd640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:14.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.081+0000 7fb7deffd640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:14.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.081+0000 7fb7deffd640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-25T18:17:14.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.082+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb7d00503b0 con 0x7fb7e8146060 2024-03-25T18:17:14.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.082+0000 7fb7deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:14.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.082+0000 7fb7deffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:14.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.300+0000 7fb7ecdcd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:14.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.300+0000 7fb7ecdcd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:14.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.300+0000 7fb7ecdcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:17:14.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.300+0000 7fb7ecdcd640 1 -- 172.21.15.64:0/171944699 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7e80adb60 con 0x7fb7e8146060 2024-03-25T18:17:14.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.394+0000 7fb7deffd640 1 -- 172.21.15.64:0/171944699 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fb7e80adb60 con 0x7fb7e8146060 2024-03-25T18:17:14.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.394+0000 7fb7deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:14.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.394+0000 7fb7deffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:14.395 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:17:14.396 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:14.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.396+0000 7fb7ecdcd640 10 librados: watch_flush enter 2024-03-25T18:17:14.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.396+0000 7fb7ecdcd640 10 librados: watch_flush exit 2024-03-25T18:17:14.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.396+0000 7fb7ecdcd640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-25T18:17:14.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.396+0000 7fb7ecdcd640 10 client.4571.objecter successfully canceled tick 2024-03-25T18:17:14.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.396+0000 7fb7ecdcd640 1 -- 172.21.15.64:0/171944699 >> v1:172.21.15.64:6800/2915543638 conn(0x7fb7c806ddf0 legacy=0x7fb7c80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:14.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.396+0000 7fb7ecdcd640 10 monclient: shutdown 2024-03-25T18:17:14.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.396+0000 7fb7ecdcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:14.398 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.396+0000 7fb7ecdcd640 1 -- 172.21.15.64:0/171944699 >> v1:172.21.15.64:6790/0 conn(0x7fb7e8146060 legacy=0x7fb7e80af2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:14.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.397+0000 7fb7ecdcd640 1 -- 172.21.15.64:0/171944699 shutdown_connections 2024-03-25T18:17:14.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.397+0000 7fb7ecdcd640 1 -- 172.21.15.64:0/171944699 >> 172.21.15.64:0/171944699 conn(0x7fb7e80b7ea0 msgr2=0x7fb7e80b6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:14.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.397+0000 7fb7ecdcd640 1 -- 172.21.15.64:0/171944699 shutdown_connections 2024-03-25T18:17:14.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.397+0000 7fb7ecdcd640 1 -- 172.21.15.64:0/171944699 wait complete. 2024-03-25T18:17:14.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:14.397+0000 7fb7ecdcd640 1 librados: shutdown 2024-03-25T18:17:14.410 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:21.217 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:21.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.330+0000 7fcafdc06640 10 monclient: get_monmap_and_config 2024-03-25T18:17:21.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.330+0000 7fcafdc06640 10 monclient: build_initial_monmap 2024-03-25T18:17:21.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.330+0000 7fcafdc06640 10 monclient: monmap: 2024-03-25T18:17:21.332 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:17:21.332 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:21.332 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:17:21.331269+0000 2024-03-25T18:17:21.332 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:17:21.331269+0000 2024-03-25T18:17:21.332 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:21.332 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:21.332 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:17:21.332 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:17:21.332 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:17:21.332 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:21.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.331+0000 7fcafdc06640 1 Processor -- start 2024-03-25T18:17:21.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.331+0000 7fcafdc06640 1 -- start start 2024-03-25T18:17:21.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.331+0000 7fcafdc06640 10 monclient: init 2024-03-25T18:17:21.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.332+0000 7fcafdc06640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:21.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.332+0000 7fcafdc06640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:21.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.332+0000 7fcafdc06640 10 monclient(hunting): picked mon.noname-a con 0x7fcaf80b4d60 addr v1:172.21.15.64:6789/0 2024-03-25T18:17:21.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.332+0000 7fcafdc06640 10 monclient(hunting): picked mon.noname-c con 0x7fcaf80b1730 addr v1:172.21.15.86:6789/0 2024-03-25T18:17:21.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.332+0000 7fcafdc06640 10 monclient(hunting): picked mon.noname-b con 0x7fcaf814be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:17:21.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.332+0000 7fcafdc06640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf80af940 con 0x7fcaf80b4d60 2024-03-25T18:17:21.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.332+0000 7fcafdc06640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf80b0a70 con 0x7fcaf80b1730 2024-03-25T18:17:21.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.332+0000 7fcafdc06640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf80b0d50 con 0x7fcaf814be90 2024-03-25T18:17:21.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.332+0000 7fcafdc06640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:21.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.332+0000 7fcafdc06640 10 monclient(hunting): authenticate will time out at 7427.570710s 2024-03-25T18:17:21.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.340+0000 7fcaf77fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fcaf80b4d60 0x7fcaf813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:57816/0 (socket says 172.21.15.64:57816) 2024-03-25T18:17:21.341 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.340+0000 7fcaf77fe640 1 -- 172.21.15.64:0/3171322611 learned_addr learned my addr 172.21.15.64:0/3171322611 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:17:21.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.353+0000 7fcaf67fc640 1 -- 172.21.15.64:0/3171322611 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228398172 0 0) 0x7fcaf80b0d50 con 0x7fcaf814be90 2024-03-25T18:17:21.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.353+0000 7fcaf67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:21.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.353+0000 7fcaf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:21.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.353+0000 7fcaf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:21.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.353+0000 7fcaf67fc640 10 monclient(hunting): my global_id is 4574 2024-03-25T18:17:21.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.353+0000 7fcaf67fc640 1 -- 172.21.15.64:0/3171322611 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcae0003520 con 0x7fcaf814be90 2024-03-25T18:17:21.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.356+0000 7fcaf67fc640 1 -- 172.21.15.64:0/3171322611 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996830209 0 0) 0x7fcaf80b0a70 con 0x7fcaf80b1730 2024-03-25T18:17:21.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.356+0000 7fcaf67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:21.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.356+0000 7fcaf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:21.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.356+0000 7fcaf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:21.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.356+0000 7fcaf67fc640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:17:21.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.356+0000 7fcaf67fc640 1 -- 172.21.15.64:0/3171322611 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf80b0d50 con 0x7fcaf80b1730 2024-03-25T18:17:21.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.362+0000 7fcaf67fc640 1 -- 172.21.15.64:0/3171322611 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688661171 0 0) 0x7fcaf80af940 con 0x7fcaf80b4d60 2024-03-25T18:17:21.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.362+0000 7fcaf67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:21.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.362+0000 7fcaf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:21.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.362+0000 7fcaf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:21.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.362+0000 7fcaf67fc640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:17:21.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.362+0000 7fcaf67fc640 1 -- 172.21.15.64:0/3171322611 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf80b0a70 con 0x7fcaf80b4d60 2024-03-25T18:17:21.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.368+0000 7fcaf67fc640 1 -- 172.21.15.64:0/3171322611 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3259336357 0 0) 0x7fcae0003520 con 0x7fcaf814be90 2024-03-25T18:17:21.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.368+0000 7fcaf67fc640 1 -- 172.21.15.64:0/3171322611 >> v1:172.21.15.86:6789/0 conn(0x7fcaf80b1730 legacy=0x7fcaf80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.368+0000 7fcaf67fc640 1 -- 172.21.15.64:0/3171322611 >> v1:172.21.15.64:6789/0 conn(0x7fcaf80b4d60 legacy=0x7fcaf813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.368+0000 7fcaf67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:21.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.368+0000 7fcaf67fc640 1 monclient: found mon.noname-b 2024-03-25T18:17:21.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.368+0000 7fcaf67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:21.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.368+0000 7fcaf67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:17:21.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.368+0000 7fcaf67fc640 1 -- 172.21.15.64:0/3171322611 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf80b1030 con 0x7fcaf814be90 2024-03-25T18:17:21.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.368+0000 7fcaf67fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:21.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.368+0000 7fcaf67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:21.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.368+0000 7fcaf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:21.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.368+0000 7fcaf67fc640 1 -- 172.21.15.64:0/3171322611 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcaf0001a10 con 0x7fcaf814be90 2024-03-25T18:17:21.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.368+0000 7fcafdc06640 5 monclient: authenticate success, global_id 4574 2024-03-25T18:17:21.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.368+0000 7fcafdc06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:21.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.368+0000 7fcaf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:21.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.368+0000 7fcaf67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:21.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.368+0000 7fcaf67fc640 10 monclient: dump: 2024-03-25T18:17:21.370 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:21.370 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:21.371 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:21.371 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:21.371 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:21.371 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:21.371 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:21.371 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:21.371 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:21.371 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:21.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.368+0000 7fcafdc06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:21.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.375+0000 7fcaf67fc640 1 -- 172.21.15.64:0/3171322611 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcaf0001b70 con 0x7fcaf814be90 2024-03-25T18:17:21.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.375+0000 7fcaf67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:21.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.375+0000 7fcaf67fc640 1 -- 172.21.15.64:0/3171322611 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcaf0001f80 con 0x7fcaf814be90 2024-03-25T18:17:21.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.375+0000 7fcaf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:21.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.375+0000 7fcaf67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:21.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.375+0000 7fcaf67fc640 10 monclient: dump: 2024-03-25T18:17:21.376 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:21.376 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:21.377 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:21.377 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:21.377 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:21.377 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:21.377 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:21.377 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:21.377 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:21.377 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:21.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.375+0000 7fcafdc06640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:21.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.375+0000 7fcafdc06640 4 set_mon_vals no callback set 2024-03-25T18:17:21.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.376+0000 7fcafdc06640 10 monclient: shutdown 2024-03-25T18:17:21.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.376+0000 7fcafdc06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:21.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.376+0000 7fcafdc06640 1 -- 172.21.15.64:0/3171322611 >> v1:172.21.15.64:6790/0 conn(0x7fcaf814be90 legacy=0x7fcaf814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.376+0000 7fcafdc06640 1 -- 172.21.15.64:0/3171322611 shutdown_connections 2024-03-25T18:17:21.378 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.376+0000 7fcafdc06640 1 -- 172.21.15.64:0/3171322611 >> 172.21.15.64:0/3171322611 conn(0x7fcaf80617a0 msgr2=0x7fcaf8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:21.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.376+0000 7fcafdc06640 1 -- 172.21.15.64:0/3171322611 shutdown_connections 2024-03-25T18:17:21.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.376+0000 7fcafdc06640 1 -- 172.21.15.64:0/3171322611 wait complete. 2024-03-25T18:17:21.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.377+0000 7fcafdc06640 10 monclient: build_initial_monmap 2024-03-25T18:17:21.379 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.377+0000 7fcafdc06640 10 monclient: monmap: 2024-03-25T18:17:21.379 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:17:21.379 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:21.379 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:17:21.379 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:17:21.379 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:21.379 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:17:21.379 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:17:21.379 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:17:21.379 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:17:21.380 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:21.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.378+0000 7fcafdc06640 1 librados: starting msgr at 2024-03-25T18:17:21.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.378+0000 7fcafdc06640 1 librados: starting objecter 2024-03-25T18:17:21.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.378+0000 7fcafdc06640 1 Processor -- start 2024-03-25T18:17:21.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.378+0000 7fcafdc06640 1 -- start start 2024-03-25T18:17:21.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.378+0000 7fcafdc06640 1 librados: setting wanted keys 2024-03-25T18:17:21.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.378+0000 7fcafdc06640 1 librados: calling monclient init 2024-03-25T18:17:21.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.378+0000 7fcafdc06640 10 monclient: init 2024-03-25T18:17:21.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.379+0000 7fcafdc06640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:21.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.379+0000 7fcafdc06640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:21.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.379+0000 7fcafdc06640 10 monclient(hunting): picked mon.noname-b con 0x7fcaf80b1730 addr v1:172.21.15.86:6789/0 2024-03-25T18:17:21.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.379+0000 7fcafdc06640 10 monclient(hunting): picked mon.noname-c con 0x7fcaf80b4d60 addr v1:172.21.15.64:6790/0 2024-03-25T18:17:21.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.379+0000 7fcafdc06640 10 monclient(hunting): picked mon.noname-a con 0x7fcaf814be90 addr v1:172.21.15.64:6789/0 2024-03-25T18:17:21.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.379+0000 7fcafdc06640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf8138490 con 0x7fcaf814be90 2024-03-25T18:17:21.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.379+0000 7fcafdc06640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf8136cc0 con 0x7fcaf80b1730 2024-03-25T18:17:21.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.379+0000 7fcafdc06640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf8140520 con 0x7fcaf80b4d60 2024-03-25T18:17:21.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.379+0000 7fcafdc06640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:21.383 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.379+0000 7fcafdc06640 10 monclient(hunting): authenticate will time out at 7427.618104s 2024-03-25T18:17:21.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.389+0000 7fcaf7fff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fcaf814be90 0x7fcaf81e2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:57864/0 (socket says 172.21.15.64:57864) 2024-03-25T18:17:21.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.389+0000 7fcaf7fff640 1 -- 172.21.15.64:0/4795471 learned_addr learned my addr 172.21.15.64:0/4795471 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:17:21.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.392+0000 7fcadffff640 10 client.?.objecter ms_handle_connect 0x7fcaf814be90 2024-03-25T18:17:21.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.392+0000 7fcadffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:21.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.403+0000 7fcadffff640 1 -- 172.21.15.64:0/4795471 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862830562 0 0) 0x7fcaf8138490 con 0x7fcaf814be90 2024-03-25T18:17:21.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.403+0000 7fcadffff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:21.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.403+0000 7fcadffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:21.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.403+0000 7fcadffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:21.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.403+0000 7fcadffff640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:17:21.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.403+0000 7fcadffff640 1 -- 172.21.15.64:0/4795471 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcae4004eb0 con 0x7fcaf814be90 2024-03-25T18:17:21.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.405+0000 7fcadffff640 10 client.?.objecter ms_handle_connect 0x7fcaf80b1730 2024-03-25T18:17:21.406 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.405+0000 7fcadffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:21.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.418+0000 7fcadffff640 1 -- 172.21.15.64:0/4795471 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885277405 0 0) 0x7fcaf8136cc0 con 0x7fcaf80b1730 2024-03-25T18:17:21.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.418+0000 7fcadffff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:21.419 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.418+0000 7fcadffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:21.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.418+0000 7fcadffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:21.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.418+0000 7fcadffff640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:17:21.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.418+0000 7fcadffff640 1 -- 172.21.15.64:0/4795471 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf8138490 con 0x7fcaf80b1730 2024-03-25T18:17:21.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.418+0000 7fcadffff640 1 -- 172.21.15.64:0/4795471 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2527488937 0 0) 0x7fcae4004eb0 con 0x7fcaf814be90 2024-03-25T18:17:21.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.418+0000 7fcadffff640 1 -- 172.21.15.64:0/4795471 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcaf8136cc0 con 0x7fcaf814be90 2024-03-25T18:17:21.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.418+0000 7fcadffff640 1 -- 172.21.15.64:0/4795471 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcaf0001f40 con 0x7fcaf814be90 2024-03-25T18:17:21.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.418+0000 7fcadffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:21.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.418+0000 7fcadffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:21.420 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.418+0000 7fcadffff640 10 monclient(hunting): dump: 2024-03-25T18:17:21.420 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:21.420 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:21.420 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:21.420 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:21.420 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:21.421 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:21.421 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:21.421 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:21.421 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:21.421 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:21.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.418+0000 7fcadffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:21.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.423+0000 7fcadffff640 10 client.?.objecter ms_handle_connect 0x7fcaf80b4d60 2024-03-25T18:17:21.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.423+0000 7fcadffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:21.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcadffff640 1 -- 172.21.15.64:0/4795471 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2100074153 0 0) 0x7fcaf8136cc0 con 0x7fcaf814be90 2024-03-25T18:17:21.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcadffff640 1 -- 172.21.15.64:0/4795471 >> v1:172.21.15.64:6790/0 conn(0x7fcaf80b4d60 legacy=0x7fcaf8144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcadffff640 1 -- 172.21.15.64:0/4795471 >> v1:172.21.15.86:6789/0 conn(0x7fcaf80b1730 legacy=0x7fcaf8140940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcadffff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:21.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcadffff640 1 monclient: found mon.a 2024-03-25T18:17:21.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcadffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:21.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcadffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:17:21.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcadffff640 1 -- 172.21.15.64:0/4795471 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf8138700 con 0x7fcaf814be90 2024-03-25T18:17:21.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcadffff640 10 monclient: _finish_auth 0 2024-03-25T18:17:21.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcadffff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:21.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcadffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:21.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcafdc06640 5 monclient: authenticate success, global_id 4575 2024-03-25T18:17:21.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcafdc06640 10 monclient: _renew_subs 2024-03-25T18:17:21.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcafdc06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:17:21.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcafdc06640 1 -- 172.21.15.64:0/4795471 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf8136660 con 0x7fcaf814be90 2024-03-25T18:17:21.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcafdc06640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:21.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcafdc06640 10 monclient: _renew_subs 2024-03-25T18:17:21.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcafdc06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:17:21.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcafdc06640 1 -- 172.21.15.64:0/4795471 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf8137350 con 0x7fcaf814be90 2024-03-25T18:17:21.432 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.429+0000 7fcafdc06640 1 librados: init done 2024-03-25T18:17:21.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.430+0000 7fcafdc06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:21.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.430+0000 7fcafdc06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:21.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.430+0000 7fcafdc06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:17:21.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.430+0000 7fcafdc06640 1 -- 172.21.15.64:0/4795471 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf81376a0 con 0x7fcaf814be90 2024-03-25T18:17:21.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.726+0000 7fcadffff640 1 -- 172.21.15.64:0/4795471 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcaf0001510 con 0x7fcaf814be90 2024-03-25T18:17:21.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.726+0000 7fcadffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:21.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.726+0000 7fcaf5ffb640 4 set_mon_vals no callback set 2024-03-25T18:17:21.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.738+0000 7fcadffff640 1 -- 172.21.15.64:0/4795471 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcaf0003dc0 con 0x7fcaf814be90 2024-03-25T18:17:21.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.738+0000 7fcadffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:21.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.738+0000 7fcadffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:21.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.738+0000 7fcadffff640 10 monclient: dump: 2024-03-25T18:17:21.739 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:21.739 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:21.739 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:21.739 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:21.739 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:21.739 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:21.739 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:21.739 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:21.740 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:21.740 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:21.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.739+0000 7fcadffff640 1 -- 172.21.15.64:0/4795471 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fcaf0003fa0 con 0x7fcaf814be90 2024-03-25T18:17:21.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.739+0000 7fcaf77fe640 10 monclient: get_auth_request con 0x7fcae406dcd0 auth_method 0 2024-03-25T18:17:21.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.739+0000 7fcadffff640 10 client.4575.objecter ms_handle_connect 0x7fcae406dcd0 2024-03-25T18:17:21.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.744+0000 7fcadffff640 1 -- 172.21.15.64:0/4795471 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fcaf0054b70 con 0x7fcaf814be90 2024-03-25T18:17:21.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.744+0000 7fcadffff640 10 client.4575.objecter ms_dispatch 0x7fcaf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:21.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.744+0000 7fcadffff640 3 client.4575.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:21.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.744+0000 7fcadffff640 3 client.4575.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:21.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.744+0000 7fcadffff640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-25T18:17:21.745 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.744+0000 7fcadffff640 1 -- 172.21.15.64:0/4795471 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcaf00504b0 con 0x7fcaf814be90 2024-03-25T18:17:21.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.744+0000 7fcadffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:21.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.744+0000 7fcadffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:21.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.989+0000 7fcafdc06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:21.990 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.989+0000 7fcafdc06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:21.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.989+0000 7fcafdc06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:17:21.991 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:21.989+0000 7fcafdc06640 1 -- 172.21.15.64:0/4795471 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcaf80627b0 con 0x7fcaf814be90 2024-03-25T18:17:22.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:22.086+0000 7fcadffff640 1 -- 172.21.15.64:0/4795471 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fcaf00525b0 con 0x7fcaf814be90 2024-03-25T18:17:22.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:22.086+0000 7fcadffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:22.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:22.086+0000 7fcadffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:22.087 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:17:22.087 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:22.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:22.088+0000 7fcafdc06640 10 librados: watch_flush enter 2024-03-25T18:17:22.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:22.088+0000 7fcafdc06640 10 librados: watch_flush exit 2024-03-25T18:17:22.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:22.088+0000 7fcafdc06640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-25T18:17:22.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:22.088+0000 7fcafdc06640 10 client.4575.objecter successfully canceled tick 2024-03-25T18:17:22.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:22.088+0000 7fcafdc06640 1 -- 172.21.15.64:0/4795471 >> v1:172.21.15.64:6800/2915543638 conn(0x7fcae406dcd0 legacy=0x7fcae4070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:22.089 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:22.088+0000 7fcafdc06640 10 monclient: shutdown 2024-03-25T18:17:22.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:22.088+0000 7fcafdc06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:22.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:22.088+0000 7fcafdc06640 1 -- 172.21.15.64:0/4795471 >> v1:172.21.15.64:6789/0 conn(0x7fcaf814be90 legacy=0x7fcaf81e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:22.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:22.088+0000 7fcafdc06640 1 -- 172.21.15.64:0/4795471 shutdown_connections 2024-03-25T18:17:22.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:22.088+0000 7fcafdc06640 1 -- 172.21.15.64:0/4795471 >> 172.21.15.64:0/4795471 conn(0x7fcaf80617a0 msgr2=0x7fcaf80b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:22.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:22.088+0000 7fcafdc06640 1 -- 172.21.15.64:0/4795471 shutdown_connections 2024-03-25T18:17:22.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:22.088+0000 7fcafdc06640 1 -- 172.21.15.64:0/4795471 wait complete. 2024-03-25T18:17:22.090 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:22.088+0000 7fcafdc06640 1 librados: shutdown 2024-03-25T18:17:22.100 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:22.853 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:08:05.965 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:22.863 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:08:05.975 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:22.890 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:08:06.003 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:22.913 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:08:06.026 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:22.915 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:08:05.994 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:22.926 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:08:06.004 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:22.928 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:08:06.040 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:22.957 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:08:06.036 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:22.974 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:08:06.086 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:28.906 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:29.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.006+0000 7fcd51a0f640 10 monclient: get_monmap_and_config 2024-03-25T18:17:29.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.006+0000 7fcd51a0f640 10 monclient: build_initial_monmap 2024-03-25T18:17:29.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.006+0000 7fcd51a0f640 10 monclient: monmap: 2024-03-25T18:17:29.008 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:17:29.008 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:29.009 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:17:29.007833+0000 2024-03-25T18:17:29.009 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:17:29.007833+0000 2024-03-25T18:17:29.009 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:29.009 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:29.009 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:17:29.009 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:17:29.009 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:17:29.009 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:29.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.007+0000 7fcd51a0f640 1 Processor -- start 2024-03-25T18:17:29.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.007+0000 7fcd51a0f640 1 -- start start 2024-03-25T18:17:29.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.007+0000 7fcd51a0f640 10 monclient: init 2024-03-25T18:17:29.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.008+0000 7fcd51a0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:29.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.008+0000 7fcd51a0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:29.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.008+0000 7fcd51a0f640 10 monclient(hunting): picked mon.noname-b con 0x7fcd4c0a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:17:29.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.008+0000 7fcd51a0f640 10 monclient(hunting): picked mon.noname-c con 0x7fcd4c13f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:17:29.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.008+0000 7fcd51a0f640 10 monclient(hunting): picked mon.noname-a con 0x7fcd4c1430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:17:29.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.008+0000 7fcd51a0f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd4c142520 con 0x7fcd4c1430c0 2024-03-25T18:17:29.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.008+0000 7fcd51a0f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd4c0b28a0 con 0x7fcd4c13f710 2024-03-25T18:17:29.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.008+0000 7fcd51a0f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd4c0b2a30 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.008+0000 7fcd51a0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:29.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.008+0000 7fcd51a0f640 10 monclient(hunting): authenticate will time out at 7435.246723s 2024-03-25T18:17:29.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.019+0000 7fcd4a7fc640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fcd4c13f710 0x7fcd4c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:35298/0 (socket says 172.21.15.64:35298) 2024-03-25T18:17:29.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.019+0000 7fcd4a7fc640 1 -- 172.21.15.64:0/4118318769 learned_addr learned my addr 172.21.15.64:0/4118318769 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:17:29.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.035+0000 7fcd49ffb640 1 -- 172.21.15.64:0/4118318769 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618543087 0 0) 0x7fcd4c0b28a0 con 0x7fcd4c13f710 2024-03-25T18:17:29.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.035+0000 7fcd49ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:29.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.035+0000 7fcd49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:29.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.035+0000 7fcd49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:29.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.035+0000 7fcd49ffb640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:17:29.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.035+0000 7fcd49ffb640 1 -- 172.21.15.64:0/4118318769 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd34003520 con 0x7fcd4c13f710 2024-03-25T18:17:29.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.040+0000 7fcd49ffb640 1 -- 172.21.15.64:0/4118318769 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441561919 0 0) 0x7fcd4c0b2a30 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.040+0000 7fcd49ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:29.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.040+0000 7fcd49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:29.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.040+0000 7fcd49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:29.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.040+0000 7fcd49ffb640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:17:29.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.040+0000 7fcd49ffb640 1 -- 172.21.15.64:0/4118318769 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd4c0b28a0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.051+0000 7fcd49ffb640 1 -- 172.21.15.64:0/4118318769 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3299708410 0 0) 0x7fcd4c0b28a0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.051+0000 7fcd49ffb640 1 -- 172.21.15.64:0/4118318769 >> v1:172.21.15.86:6789/0 conn(0x7fcd4c13f710 legacy=0x7fcd4c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.051+0000 7fcd49ffb640 1 -- 172.21.15.64:0/4118318769 >> v1:172.21.15.64:6789/0 conn(0x7fcd4c1430c0 legacy=0x7fcd4c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.051+0000 7fcd49ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:29.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.051+0000 7fcd49ffb640 1 monclient: found mon.noname-b 2024-03-25T18:17:29.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.051+0000 7fcd49ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:29.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.051+0000 7fcd49ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:17:29.052 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.051+0000 7fcd49ffb640 1 -- 172.21.15.64:0/4118318769 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd4c0b44c0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.051+0000 7fcd49ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:29.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.051+0000 7fcd49ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:29.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.051+0000 7fcd49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:29.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.051+0000 7fcd49ffb640 1 -- 172.21.15.64:0/4118318769 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcd3c001670 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.051+0000 7fcd49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:29.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.051+0000 7fcd49ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:29.053 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.051+0000 7fcd49ffb640 10 monclient: dump: 2024-03-25T18:17:29.053 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:29.053 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:29.053 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:29.053 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:29.053 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:29.053 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:29.053 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:29.053 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:29.053 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:29.053 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:29.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.051+0000 7fcd51a0f640 5 monclient: authenticate success, global_id 4580 2024-03-25T18:17:29.054 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.051+0000 7fcd51a0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:29.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.067+0000 7fcd49ffb640 1 -- 172.21.15.64:0/4118318769 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd3c0017d0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.067+0000 7fcd49ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:29.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.067+0000 7fcd51a0f640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:29.069 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.068+0000 7fcd51a0f640 4 set_mon_vals no callback set 2024-03-25T18:17:29.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.068+0000 7fcd51a0f640 10 monclient: shutdown 2024-03-25T18:17:29.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.068+0000 7fcd51a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:29.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.068+0000 7fcd51a0f640 1 -- 172.21.15.64:0/4118318769 >> v1:172.21.15.64:6790/0 conn(0x7fcd4c0a98f0 legacy=0x7fcd4c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.068+0000 7fcd51a0f640 1 -- 172.21.15.64:0/4118318769 shutdown_connections 2024-03-25T18:17:29.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.068+0000 7fcd51a0f640 1 -- 172.21.15.64:0/4118318769 >> 172.21.15.64:0/4118318769 conn(0x7fcd4c0b7ea0 msgr2=0x7fcd4c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:29.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.069+0000 7fcd51a0f640 1 -- 172.21.15.64:0/4118318769 shutdown_connections 2024-03-25T18:17:29.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.069+0000 7fcd51a0f640 1 -- 172.21.15.64:0/4118318769 wait complete. 2024-03-25T18:17:29.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.069+0000 7fcd51a0f640 10 monclient: build_initial_monmap 2024-03-25T18:17:29.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.069+0000 7fcd51a0f640 10 monclient: monmap: 2024-03-25T18:17:29.071 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:17:29.071 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:29.071 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:17:29.071 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:17:29.071 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:29.071 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:17:29.072 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:17:29.072 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:17:29.072 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:17:29.072 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:29.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.070+0000 7fcd51a0f640 1 librados: starting msgr at 2024-03-25T18:17:29.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.070+0000 7fcd51a0f640 1 librados: starting objecter 2024-03-25T18:17:29.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.070+0000 7fcd51a0f640 1 Processor -- start 2024-03-25T18:17:29.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.070+0000 7fcd51a0f640 1 -- start start 2024-03-25T18:17:29.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.070+0000 7fcd51a0f640 1 librados: setting wanted keys 2024-03-25T18:17:29.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.070+0000 7fcd51a0f640 1 librados: calling monclient init 2024-03-25T18:17:29.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.070+0000 7fcd51a0f640 10 monclient: init 2024-03-25T18:17:29.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.071+0000 7fcd51a0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:29.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.071+0000 7fcd51a0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:29.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.071+0000 7fcd51a0f640 10 monclient(hunting): picked mon.noname-b con 0x7fcd4c0a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:17:29.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.071+0000 7fcd51a0f640 10 monclient(hunting): picked mon.noname-c con 0x7fcd4c13f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:17:29.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.071+0000 7fcd51a0f640 10 monclient(hunting): picked mon.noname-a con 0x7fcd4c1430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:17:29.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.071+0000 7fcd51a0f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd4c0abc50 con 0x7fcd4c1430c0 2024-03-25T18:17:29.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.071+0000 7fcd51a0f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd4c1d72f0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.071+0000 7fcd51a0f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd4c1d8170 con 0x7fcd4c13f710 2024-03-25T18:17:29.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.071+0000 7fcd51a0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:29.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.071+0000 7fcd51a0f640 10 monclient(hunting): authenticate will time out at 7435.310036s 2024-03-25T18:17:29.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.081+0000 7fcd4affd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fcd4c0a98f0 0x7fcd4c1d7a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:35390/0 (socket says 172.21.15.64:35390) 2024-03-25T18:17:29.082 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.081+0000 7fcd4affd640 1 -- 172.21.15.64:0/183140966 learned_addr learned my addr 172.21.15.64:0/183140966 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:17:29.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.082+0000 7fcd33fff640 10 client.?.objecter ms_handle_connect 0x7fcd4c0a98f0 2024-03-25T18:17:29.083 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.082+0000 7fcd33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:29.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.094+0000 7fcd33fff640 10 client.?.objecter ms_handle_connect 0x7fcd4c1430c0 2024-03-25T18:17:29.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.094+0000 7fcd33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:29.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.099+0000 7fcd33fff640 10 client.?.objecter ms_handle_connect 0x7fcd4c13f710 2024-03-25T18:17:29.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.099+0000 7fcd33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:29.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.099+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754348826 0 0) 0x7fcd4c1d72f0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.099+0000 7fcd33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:29.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.099+0000 7fcd33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:29.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.099+0000 7fcd33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:29.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.099+0000 7fcd33fff640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:17:29.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.099+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd38004eb0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.112 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.112+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 373617963 0 0) 0x7fcd38004eb0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.112+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd4c1d72f0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.112+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcd3c0035f0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.112+0000 7fcd33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:29.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.112+0000 7fcd33fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:29.113 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.112+0000 7fcd33fff640 10 monclient(hunting): dump: 2024-03-25T18:17:29.113 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:29.113 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:29.113 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:29.113 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:29.113 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:29.113 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:29.113 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:29.113 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:29.113 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:29.113 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:29.114 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.112+0000 7fcd33fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:29.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.118+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248735389 0 0) 0x7fcd4c1d8170 con 0x7fcd4c13f710 2024-03-25T18:17:29.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.118+0000 7fcd33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:29.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.118+0000 7fcd33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:29.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.118+0000 7fcd33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:29.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.118+0000 7fcd33fff640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:17:29.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.118+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd38004eb0 con 0x7fcd4c13f710 2024-03-25T18:17:29.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.120+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267983947 0 0) 0x7fcd4c0abc50 con 0x7fcd4c1430c0 2024-03-25T18:17:29.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.120+0000 7fcd33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:29.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.120+0000 7fcd33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:29.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.120+0000 7fcd33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:29.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.120+0000 7fcd33fff640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:17:29.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.120+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd4c1d8170 con 0x7fcd4c1430c0 2024-03-25T18:17:29.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2755664183 0 0) 0x7fcd4c1d72f0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 >> v1:172.21.15.64:6790/0 conn(0x7fcd4c13f710 legacy=0x7fcd4c133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 >> v1:172.21.15.64:6789/0 conn(0x7fcd4c1430c0 legacy=0x7fcd4c1e2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd33fff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:29.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd33fff640 1 monclient: found mon.b 2024-03-25T18:17:29.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:29.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd33fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:17:29.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd4c1378d0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd33fff640 10 monclient: _finish_auth 0 2024-03-25T18:17:29.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd33fff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:29.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:29.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd51a0f640 5 monclient: authenticate success, global_id 4582 2024-03-25T18:17:29.127 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd51a0f640 10 monclient: _renew_subs 2024-03-25T18:17:29.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd51a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:17:29.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd51a0f640 1 -- 172.21.15.64:0/183140966 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd4c139970 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd51a0f640 10 client.4582.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:29.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd51a0f640 10 monclient: _renew_subs 2024-03-25T18:17:29.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd51a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:17:29.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd51a0f640 1 -- 172.21.15.64:0/183140966 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd4c138e80 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.125+0000 7fcd51a0f640 1 librados: init done 2024-03-25T18:17:29.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.127+0000 7fcd51a0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:29.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.127+0000 7fcd51a0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:29.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.127+0000 7fcd51a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:17:29.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.127+0000 7fcd51a0f640 1 -- 172.21.15.64:0/183140966 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd4c1d78b0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.402+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd3c003a00 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.402+0000 7fcd33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:29.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.402+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcd3c003dd0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.402+0000 7fcd33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:29.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.402+0000 7fcd33fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:29.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.402+0000 7fcd33fff640 10 monclient: dump: 2024-03-25T18:17:29.404 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:29.404 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:29.404 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:29.404 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:29.404 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:29.404 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:29.404 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:29.404 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:29.404 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:29.404 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:29.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.402+0000 7fcd497fa640 4 set_mon_vals no callback set 2024-03-25T18:17:29.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.403+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fcd3c003ff0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.403+0000 7fcd4a7fc640 10 monclient: get_auth_request con 0x7fcd3806dd70 auth_method 0 2024-03-25T18:17:29.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.404+0000 7fcd33fff640 10 client.4582.objecter ms_handle_connect 0x7fcd3806dd70 2024-03-25T18:17:29.408 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.407+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fcd3c054aa0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.407+0000 7fcd33fff640 10 client.4582.objecter ms_dispatch 0x7fcd4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:29.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.407+0000 7fcd33fff640 3 client.4582.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:29.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.407+0000 7fcd33fff640 3 client.4582.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:29.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.407+0000 7fcd33fff640 20 client.4582.objecter dump_active .. 0 homeless 2024-03-25T18:17:29.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.408+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcd3c0505b0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.408+0000 7fcd33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:29.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.408+0000 7fcd33fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:29.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.657+0000 7fcd51a0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:29.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.657+0000 7fcd51a0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:29.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.657+0000 7fcd51a0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:17:29.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.657+0000 7fcd51a0f640 1 -- 172.21.15.64:0/183140966 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd4c0b3ef0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.689+0000 7fcd33fff640 1 -- 172.21.15.64:0/183140966 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fcd3c0524e0 con 0x7fcd4c0a98f0 2024-03-25T18:17:29.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.689+0000 7fcd33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:29.690 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.689+0000 7fcd33fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:29.690 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:17:29.691 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:29.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.691+0000 7fcd51a0f640 10 librados: watch_flush enter 2024-03-25T18:17:29.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.691+0000 7fcd51a0f640 10 librados: watch_flush exit 2024-03-25T18:17:29.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.691+0000 7fcd51a0f640 20 client.4582.objecter shutdown clearing up homeless session... 2024-03-25T18:17:29.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.691+0000 7fcd51a0f640 10 client.4582.objecter successfully canceled tick 2024-03-25T18:17:29.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.691+0000 7fcd51a0f640 1 -- 172.21.15.64:0/183140966 >> v1:172.21.15.64:6800/2915543638 conn(0x7fcd3806dd70 legacy=0x7fcd38070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.691+0000 7fcd51a0f640 10 monclient: shutdown 2024-03-25T18:17:29.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.691+0000 7fcd51a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:29.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.691+0000 7fcd51a0f640 1 -- 172.21.15.64:0/183140966 >> v1:172.21.15.86:6789/0 conn(0x7fcd4c0a98f0 legacy=0x7fcd4c1d7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.691+0000 7fcd51a0f640 1 -- 172.21.15.64:0/183140966 shutdown_connections 2024-03-25T18:17:29.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.691+0000 7fcd51a0f640 1 -- 172.21.15.64:0/183140966 >> 172.21.15.64:0/183140966 conn(0x7fcd4c0b7ea0 msgr2=0x7fcd4c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:29.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.692+0000 7fcd51a0f640 1 -- 172.21.15.64:0/183140966 shutdown_connections 2024-03-25T18:17:29.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.692+0000 7fcd51a0f640 1 -- 172.21.15.64:0/183140966 wait complete. 2024-03-25T18:17:29.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:29.692+0000 7fcd51a0f640 1 librados: shutdown 2024-03-25T18:17:29.704 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:36.511 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:36.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.625+0000 7fcc70a43640 10 monclient: get_monmap_and_config 2024-03-25T18:17:36.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.625+0000 7fcc70a43640 10 monclient: build_initial_monmap 2024-03-25T18:17:36.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.625+0000 7fcc70a43640 10 monclient: monmap: 2024-03-25T18:17:36.628 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:17:36.628 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:36.628 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:17:36.627791+0000 2024-03-25T18:17:36.628 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:17:36.627791+0000 2024-03-25T18:17:36.628 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:36.628 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:36.628 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:17:36.628 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:17:36.628 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:17:36.628 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:36.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.626+0000 7fcc70a43640 1 Processor -- start 2024-03-25T18:17:36.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.626+0000 7fcc70a43640 1 -- start start 2024-03-25T18:17:36.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.626+0000 7fcc70a43640 10 monclient: init 2024-03-25T18:17:36.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.627+0000 7fcc70a43640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:36.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.627+0000 7fcc70a43640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:36.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.627+0000 7fcc70a43640 10 monclient(hunting): picked mon.noname-b con 0x7fcc6c06c380 addr v1:172.21.15.64:6790/0 2024-03-25T18:17:36.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.627+0000 7fcc70a43640 10 monclient(hunting): picked mon.noname-c con 0x7fcc6c0b12d0 addr v1:172.21.15.86:6789/0 2024-03-25T18:17:36.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.627+0000 7fcc70a43640 10 monclient(hunting): picked mon.noname-a con 0x7fcc6c0aedf0 addr v1:172.21.15.64:6789/0 2024-03-25T18:17:36.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.627+0000 7fcc70a43640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc6c0b16f0 con 0x7fcc6c0aedf0 2024-03-25T18:17:36.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.627+0000 7fcc70a43640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc6c0adfe0 con 0x7fcc6c0b12d0 2024-03-25T18:17:36.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.627+0000 7fcc70a43640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc6c0add00 con 0x7fcc6c06c380 2024-03-25T18:17:36.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.627+0000 7fcc70a43640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:36.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.627+0000 7fcc70a43640 10 monclient(hunting): authenticate will time out at 7442.867211s 2024-03-25T18:17:36.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.631+0000 7fcc69d9b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fcc6c0b12d0 0x7fcc6c0ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:56952/0 (socket says 172.21.15.64:56952) 2024-03-25T18:17:36.633 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.631+0000 7fcc69d9b640 1 -- 172.21.15.64:0/3896678250 learned_addr learned my addr 172.21.15.64:0/3896678250 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:17:36.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.645+0000 7fcc6959a640 1 -- 172.21.15.64:0/3896678250 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510016151 0 0) 0x7fcc6c0adfe0 con 0x7fcc6c0b12d0 2024-03-25T18:17:36.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.645+0000 7fcc6959a640 10 monclient(hunting): _negotiate 2024-03-25T18:17:36.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.645+0000 7fcc6959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:36.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.645+0000 7fcc6959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:36.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.645+0000 7fcc6959a640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:17:36.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.645+0000 7fcc6959a640 1 -- 172.21.15.64:0/3896678250 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc54003520 con 0x7fcc6c0b12d0 2024-03-25T18:17:36.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.660+0000 7fcc6959a640 1 -- 172.21.15.64:0/3896678250 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2932038052 0 0) 0x7fcc54003520 con 0x7fcc6c0b12d0 2024-03-25T18:17:36.662 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.660+0000 7fcc6959a640 1 -- 172.21.15.64:0/3896678250 >> v1:172.21.15.64:6790/0 conn(0x7fcc6c06c380 legacy=0x7fcc6c0b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:17:36.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.660+0000 7fcc6959a640 1 -- 172.21.15.64:0/3896678250 >> v1:172.21.15.64:6789/0 conn(0x7fcc6c0aedf0 legacy=0x7fcc6c14e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:36.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.660+0000 7fcc6959a640 10 monclient: _finish_hunting 0 2024-03-25T18:17:36.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.660+0000 7fcc6959a640 1 monclient: found mon.noname-c 2024-03-25T18:17:36.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.660+0000 7fcc6959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:36.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.660+0000 7fcc6959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:17:36.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.660+0000 7fcc6959a640 1 -- 172.21.15.64:0/3896678250 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6c0ae2c0 con 0x7fcc6c0b12d0 2024-03-25T18:17:36.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.660+0000 7fcc6959a640 10 monclient: _finish_auth 0 2024-03-25T18:17:36.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.660+0000 7fcc6959a640 10 monclient: _check_auth_tickets 2024-03-25T18:17:36.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.660+0000 7fcc6959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:36.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.660+0000 7fcc6959a640 1 -- 172.21.15.64:0/3896678250 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcc60001690 con 0x7fcc6c0b12d0 2024-03-25T18:17:36.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.660+0000 7fcc6959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:36.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.660+0000 7fcc6959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:36.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.660+0000 7fcc6959a640 10 monclient: dump: 2024-03-25T18:17:36.663 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:36.663 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:36.664 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:36.664 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:36.664 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:36.664 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:36.664 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:36.664 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:36.664 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:36.664 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:36.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.660+0000 7fcc70a43640 5 monclient: authenticate success, global_id 4585 2024-03-25T18:17:36.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.660+0000 7fcc70a43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:36.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.680+0000 7fcc6959a640 1 -- 172.21.15.64:0/3896678250 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc600017f0 con 0x7fcc6c0b12d0 2024-03-25T18:17:36.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.680+0000 7fcc6959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:36.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.680+0000 7fcc6959a640 1 -- 172.21.15.64:0/3896678250 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcc60001c40 con 0x7fcc6c0b12d0 2024-03-25T18:17:36.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.680+0000 7fcc6959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:36.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.680+0000 7fcc6959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:36.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.680+0000 7fcc6959a640 10 monclient: dump: 2024-03-25T18:17:36.683 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:36.683 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:36.683 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:36.683 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:36.683 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:36.683 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:36.683 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:36.683 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:36.683 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:36.683 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:36.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.680+0000 7fcc70a43640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:36.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.680+0000 7fcc70a43640 4 set_mon_vals no callback set 2024-03-25T18:17:36.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.681+0000 7fcc70a43640 10 monclient: shutdown 2024-03-25T18:17:36.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.681+0000 7fcc70a43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:36.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.681+0000 7fcc70a43640 1 -- 172.21.15.64:0/3896678250 >> v1:172.21.15.86:6789/0 conn(0x7fcc6c0b12d0 legacy=0x7fcc6c0ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:36.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.681+0000 7fcc70a43640 1 -- 172.21.15.64:0/3896678250 shutdown_connections 2024-03-25T18:17:36.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.681+0000 7fcc70a43640 1 -- 172.21.15.64:0/3896678250 >> 172.21.15.64:0/3896678250 conn(0x7fcc6c0b5d10 msgr2=0x7fcc6c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:36.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.682+0000 7fcc70a43640 1 -- 172.21.15.64:0/3896678250 shutdown_connections 2024-03-25T18:17:36.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.682+0000 7fcc70a43640 1 -- 172.21.15.64:0/3896678250 wait complete. 2024-03-25T18:17:36.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.683+0000 7fcc70a43640 10 monclient: build_initial_monmap 2024-03-25T18:17:36.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.683+0000 7fcc70a43640 10 monclient: monmap: 2024-03-25T18:17:36.685 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:17:36.685 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:36.685 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:17:36.685 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:17:36.685 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:36.685 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:17:36.686 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:17:36.686 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:17:36.686 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:17:36.686 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:36.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.683+0000 7fcc70a43640 1 librados: starting msgr at 2024-03-25T18:17:36.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.683+0000 7fcc70a43640 1 librados: starting objecter 2024-03-25T18:17:36.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.683+0000 7fcc70a43640 1 Processor -- start 2024-03-25T18:17:36.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.683+0000 7fcc70a43640 1 -- start start 2024-03-25T18:17:36.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.683+0000 7fcc70a43640 1 librados: setting wanted keys 2024-03-25T18:17:36.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.683+0000 7fcc70a43640 1 librados: calling monclient init 2024-03-25T18:17:36.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.683+0000 7fcc70a43640 10 monclient: init 2024-03-25T18:17:36.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.684+0000 7fcc70a43640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:36.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.684+0000 7fcc70a43640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:36.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.684+0000 7fcc70a43640 10 monclient(hunting): picked mon.noname-c con 0x7fcc6c06c380 addr v1:172.21.15.64:6790/0 2024-03-25T18:17:36.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.684+0000 7fcc70a43640 10 monclient(hunting): picked mon.noname-b con 0x7fcc6c0aedf0 addr v1:172.21.15.86:6789/0 2024-03-25T18:17:36.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.684+0000 7fcc70a43640 10 monclient(hunting): picked mon.noname-a con 0x7fcc6c0b12d0 addr v1:172.21.15.64:6789/0 2024-03-25T18:17:36.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.684+0000 7fcc70a43640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc6c06c8e0 con 0x7fcc6c0b12d0 2024-03-25T18:17:36.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.684+0000 7fcc70a43640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc6c1c47d0 con 0x7fcc6c0aedf0 2024-03-25T18:17:36.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.684+0000 7fcc70a43640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc6c1d8380 con 0x7fcc6c06c380 2024-03-25T18:17:36.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.684+0000 7fcc70a43640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:36.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.684+0000 7fcc70a43640 10 monclient(hunting): authenticate will time out at 7442.924077s 2024-03-25T18:17:36.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.698+0000 7fcc6ad9d640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fcc6c0b12d0 0x7fcc6c1c2100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:47960/0 (socket says 172.21.15.64:47960) 2024-03-25T18:17:36.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.698+0000 7fcc6ad9d640 1 -- 172.21.15.64:0/2823537855 learned_addr learned my addr 172.21.15.64:0/2823537855 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:17:36.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.699+0000 7fcc5affd640 10 client.?.objecter ms_handle_connect 0x7fcc6c0b12d0 2024-03-25T18:17:36.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.699+0000 7fcc5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:36.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.702+0000 7fcc5affd640 10 client.?.objecter ms_handle_connect 0x7fcc6c0aedf0 2024-03-25T18:17:36.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.702+0000 7fcc5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:36.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.703+0000 7fcc5affd640 10 client.?.objecter ms_handle_connect 0x7fcc6c06c380 2024-03-25T18:17:36.704 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.703+0000 7fcc5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:36.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.706+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698434873 0 0) 0x7fcc6c06c8e0 con 0x7fcc6c0b12d0 2024-03-25T18:17:36.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.706+0000 7fcc5affd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:36.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.706+0000 7fcc5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:36.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.706+0000 7fcc5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:36.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.706+0000 7fcc5affd640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:17:36.709 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.706+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc4c004eb0 con 0x7fcc6c0b12d0 2024-03-25T18:17:36.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.712+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500452454 0 0) 0x7fcc6c1d8380 con 0x7fcc6c06c380 2024-03-25T18:17:36.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.712+0000 7fcc5affd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:36.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.712+0000 7fcc5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:36.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.712+0000 7fcc5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:36.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.712+0000 7fcc5affd640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:17:36.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.712+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc6c06c8e0 con 0x7fcc6c06c380 2024-03-25T18:17:36.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.712+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725583152 0 0) 0x7fcc6c1c47d0 con 0x7fcc6c0aedf0 2024-03-25T18:17:36.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.712+0000 7fcc5affd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:36.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.712+0000 7fcc5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:36.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.712+0000 7fcc5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:36.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.712+0000 7fcc5affd640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:17:36.715 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.712+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc6c1d8380 con 0x7fcc6c0aedf0 2024-03-25T18:17:36.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.716+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 317024728 0 0) 0x7fcc4c004eb0 con 0x7fcc6c0b12d0 2024-03-25T18:17:36.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.716+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc6c1c47d0 con 0x7fcc6c0b12d0 2024-03-25T18:17:36.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.716+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcc64001d30 con 0x7fcc6c0b12d0 2024-03-25T18:17:36.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.716+0000 7fcc5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:36.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.716+0000 7fcc5affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:36.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.716+0000 7fcc5affd640 10 monclient(hunting): dump: 2024-03-25T18:17:36.719 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:36.719 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:36.719 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:36.719 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:36.719 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:36.719 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:36.719 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:36.719 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:36.719 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:36.719 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:36.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.716+0000 7fcc5affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:36.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.720+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1321548933 0 0) 0x7fcc6c06c8e0 con 0x7fcc6c06c380 2024-03-25T18:17:36.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.720+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc4c004eb0 con 0x7fcc6c06c380 2024-03-25T18:17:36.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.720+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcc5c001510 con 0x7fcc6c06c380 2024-03-25T18:17:36.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.720+0000 7fcc5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:36.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.720+0000 7fcc5affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:36.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.720+0000 7fcc5affd640 10 monclient(hunting): dump: 2024-03-25T18:17:36.723 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:36.723 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:36.723 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:36.723 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:36.723 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:36.723 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:36.723 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:36.723 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:36.723 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:36.724 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:36.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3021988995 0 0) 0x7fcc6c1c47d0 con 0x7fcc6c0b12d0 2024-03-25T18:17:36.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 >> v1:172.21.15.64:6790/0 conn(0x7fcc6c06c380 legacy=0x7fcc6c1d74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:36.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 >> v1:172.21.15.86:6789/0 conn(0x7fcc6c0aedf0 legacy=0x7fcc6c1c07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:36.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc5affd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:36.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc5affd640 1 monclient: found mon.a 2024-03-25T18:17:36.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:36.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc5affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:17:36.726 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6c1d7e60 con 0x7fcc6c0b12d0 2024-03-25T18:17:36.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc5affd640 10 monclient: _finish_auth 0 2024-03-25T18:17:36.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc5affd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:36.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:36.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc70a43640 5 monclient: authenticate success, global_id 4587 2024-03-25T18:17:36.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc70a43640 10 monclient: _renew_subs 2024-03-25T18:17:36.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc70a43640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:17:36.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc70a43640 1 -- 172.21.15.64:0/2823537855 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc6c1c0ef0 con 0x7fcc6c0b12d0 2024-03-25T18:17:36.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc70a43640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:36.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc70a43640 10 monclient: _renew_subs 2024-03-25T18:17:36.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc70a43640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:17:36.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc70a43640 1 -- 172.21.15.64:0/2823537855 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc6c1c19c0 con 0x7fcc6c0b12d0 2024-03-25T18:17:36.727 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.724+0000 7fcc70a43640 1 librados: init done 2024-03-25T18:17:36.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.725+0000 7fcc70a43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:36.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.725+0000 7fcc70a43640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:36.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.725+0000 7fcc70a43640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:17:36.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:36.725+0000 7fcc70a43640 1 -- 172.21.15.64:0/2823537855 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc6c06d790 con 0x7fcc6c0b12d0 2024-03-25T18:17:37.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.057+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc64003990 con 0x7fcc6c0b12d0 2024-03-25T18:17:37.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.057+0000 7fcc5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:37.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.057+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fcc64003b80 con 0x7fcc6c0b12d0 2024-03-25T18:17:37.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.057+0000 7fcc5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:37.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.057+0000 7fcc5affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:37.059 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.057+0000 7fcc5affd640 10 monclient: dump: 2024-03-25T18:17:37.059 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:37.060 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:37.060 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:37.060 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:37.060 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:37.060 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:37.060 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:37.060 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:37.060 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:37.060 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:37.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.057+0000 7fcc68d99640 4 set_mon_vals no callback set 2024-03-25T18:17:37.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.057+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fcc64003da0 con 0x7fcc6c0b12d0 2024-03-25T18:17:37.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.058+0000 7fcc6a59c640 10 monclient: get_auth_request con 0x7fcc4c06df70 auth_method 0 2024-03-25T18:17:37.061 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.058+0000 7fcc5affd640 10 client.4587.objecter ms_handle_connect 0x7fcc4c06df70 2024-03-25T18:17:37.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.065+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fcc64054a20 con 0x7fcc6c0b12d0 2024-03-25T18:17:37.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.065+0000 7fcc5affd640 10 client.4587.objecter ms_dispatch 0x7fcc6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:37.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.065+0000 7fcc5affd640 3 client.4587.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:37.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.065+0000 7fcc5affd640 3 client.4587.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:37.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.065+0000 7fcc5affd640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-25T18:17:37.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.065+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc64050530 con 0x7fcc6c0b12d0 2024-03-25T18:17:37.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.066+0000 7fcc5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:37.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.066+0000 7fcc5affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:37.306 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.304+0000 7fcc70a43640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:37.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.304+0000 7fcc70a43640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:37.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.304+0000 7fcc70a43640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:17:37.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.304+0000 7fcc70a43640 1 -- 172.21.15.64:0/2823537855 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc6c1d7ff0 con 0x7fcc6c0b12d0 2024-03-25T18:17:37.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.331+0000 7fcc5affd640 1 -- 172.21.15.64:0/2823537855 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fcc64052460 con 0x7fcc6c0b12d0 2024-03-25T18:17:37.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.331+0000 7fcc5affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:37.333 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.331+0000 7fcc5affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:37.333 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:17:37.333 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:37.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.333+0000 7fcc70a43640 10 librados: watch_flush enter 2024-03-25T18:17:37.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.333+0000 7fcc70a43640 10 librados: watch_flush exit 2024-03-25T18:17:37.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.333+0000 7fcc70a43640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-25T18:17:37.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.333+0000 7fcc70a43640 10 client.4587.objecter successfully canceled tick 2024-03-25T18:17:37.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.333+0000 7fcc70a43640 1 -- 172.21.15.64:0/2823537855 >> v1:172.21.15.64:6800/2915543638 conn(0x7fcc4c06df70 legacy=0x7fcc4c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:37.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.333+0000 7fcc70a43640 10 monclient: shutdown 2024-03-25T18:17:37.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.333+0000 7fcc70a43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:37.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.333+0000 7fcc70a43640 1 -- 172.21.15.64:0/2823537855 >> v1:172.21.15.64:6789/0 conn(0x7fcc6c0b12d0 legacy=0x7fcc6c1c2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:37.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.334+0000 7fcc70a43640 1 -- 172.21.15.64:0/2823537855 shutdown_connections 2024-03-25T18:17:37.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.334+0000 7fcc70a43640 1 -- 172.21.15.64:0/2823537855 >> 172.21.15.64:0/2823537855 conn(0x7fcc6c0b5d10 msgr2=0x7fcc6c1cd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:37.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.334+0000 7fcc70a43640 1 -- 172.21.15.64:0/2823537855 shutdown_connections 2024-03-25T18:17:37.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.334+0000 7fcc70a43640 1 -- 172.21.15.64:0/2823537855 wait complete. 2024-03-25T18:17:37.337 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:37.334+0000 7fcc70a43640 1 librados: shutdown 2024-03-25T18:17:37.347 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:43.020 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:08:26.132 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:43.026 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:08:26.138 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:43.066 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:08:26.144 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:43.068 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:08:26.181 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:43.079 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:08:26.158 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:43.094 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:08:26.206 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:43.099 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:08:26.211 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:43.101 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:08:26.179 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:43.125 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:08:26.238 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:44.154 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:44.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.259+0000 7fc2ea8d9640 10 monclient: get_monmap_and_config 2024-03-25T18:17:44.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.259+0000 7fc2ea8d9640 10 monclient: build_initial_monmap 2024-03-25T18:17:44.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.259+0000 7fc2ea8d9640 10 monclient: monmap: 2024-03-25T18:17:44.260 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:17:44.260 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:44.260 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:17:44.259942+0000 2024-03-25T18:17:44.260 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:17:44.259942+0000 2024-03-25T18:17:44.260 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:44.260 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:44.260 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:17:44.260 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:17:44.260 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:17:44.260 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:44.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.259+0000 7fc2ea8d9640 1 Processor -- start 2024-03-25T18:17:44.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.260+0000 7fc2ea8d9640 1 -- start start 2024-03-25T18:17:44.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.260+0000 7fc2ea8d9640 10 monclient: init 2024-03-25T18:17:44.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.260+0000 7fc2ea8d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:44.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.260+0000 7fc2ea8d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:44.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.260+0000 7fc2ea8d9640 10 monclient(hunting): picked mon.noname-c con 0x7fc2e40a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:17:44.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.260+0000 7fc2ea8d9640 10 monclient(hunting): picked mon.noname-a con 0x7fc2e413f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:17:44.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.261+0000 7fc2ea8d9640 10 monclient(hunting): picked mon.noname-b con 0x7fc2e41430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:17:44.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.261+0000 7fc2ea8d9640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e4142520 con 0x7fc2e413f710 2024-03-25T18:17:44.263 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.261+0000 7fc2ea8d9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e40b28a0 con 0x7fc2e40a98f0 2024-03-25T18:17:44.263 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.261+0000 7fc2ea8d9640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e40b2a30 con 0x7fc2e41430c0 2024-03-25T18:17:44.263 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.261+0000 7fc2ea8d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:44.263 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.261+0000 7fc2ea8d9640 10 monclient(hunting): authenticate will time out at 7450.499538s 2024-03-25T18:17:44.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.277+0000 7fc2e8e76640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fc2e41430c0 0x7fc2e414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:57300/0 (socket says 172.21.15.64:57300) 2024-03-25T18:17:44.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.277+0000 7fc2e8e76640 1 -- 172.21.15.64:0/2811844174 learned_addr learned my addr 172.21.15.64:0/2811844174 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:17:44.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.286+0000 7fc2e2ffd640 1 -- 172.21.15.64:0/2811844174 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738396167 0 0) 0x7fc2e40b2a30 con 0x7fc2e41430c0 2024-03-25T18:17:44.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.286+0000 7fc2e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.286+0000 7fc2e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.286+0000 7fc2e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.286+0000 7fc2e2ffd640 10 monclient(hunting): my global_id is 4589 2024-03-25T18:17:44.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.286+0000 7fc2e2ffd640 1 -- 172.21.15.64:0/2811844174 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2cc003520 con 0x7fc2e41430c0 2024-03-25T18:17:44.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.292+0000 7fc2e2ffd640 1 -- 172.21.15.64:0/2811844174 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90046008 0 0) 0x7fc2e4142520 con 0x7fc2e413f710 2024-03-25T18:17:44.293 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.292+0000 7fc2e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.292+0000 7fc2e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.292+0000 7fc2e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.292+0000 7fc2e2ffd640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:17:44.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.292+0000 7fc2e2ffd640 1 -- 172.21.15.64:0/2811844174 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2e40b2a30 con 0x7fc2e413f710 2024-03-25T18:17:44.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.301+0000 7fc2e2ffd640 1 -- 172.21.15.64:0/2811844174 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 130236589 0 0) 0x7fc2e40b2a30 con 0x7fc2e413f710 2024-03-25T18:17:44.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.301+0000 7fc2e2ffd640 1 -- 172.21.15.64:0/2811844174 >> v1:172.21.15.64:6790/0 conn(0x7fc2e41430c0 legacy=0x7fc2e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.301+0000 7fc2e2ffd640 1 -- 172.21.15.64:0/2811844174 >> v1:172.21.15.86:6789/0 conn(0x7fc2e40a98f0 legacy=0x7fc2e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.301+0000 7fc2e2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:44.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.301+0000 7fc2e2ffd640 1 monclient: found mon.noname-a 2024-03-25T18:17:44.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.301+0000 7fc2e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:44.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.301+0000 7fc2e2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:17:44.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.301+0000 7fc2e2ffd640 1 -- 172.21.15.64:0/2811844174 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e40b44c0 con 0x7fc2e413f710 2024-03-25T18:17:44.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.301+0000 7fc2e2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:44.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.301+0000 7fc2e2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:44.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.301+0000 7fc2e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:44.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.301+0000 7fc2e2ffd640 1 -- 172.21.15.64:0/2811844174 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fc2d8001670 con 0x7fc2e413f710 2024-03-25T18:17:44.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.301+0000 7fc2e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.301+0000 7fc2e2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:44.303 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.301+0000 7fc2e2ffd640 10 monclient: dump: 2024-03-25T18:17:44.303 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:44.303 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:44.303 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:44.303 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:44.304 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.304 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:44.304 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:44.304 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:44.304 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:44.304 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:44.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.301+0000 7fc2ea8d9640 5 monclient: authenticate success, global_id 4590 2024-03-25T18:17:44.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.301+0000 7fc2ea8d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:44.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.312+0000 7fc2e2ffd640 1 -- 172.21.15.64:0/2811844174 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2d80017d0 con 0x7fc2e413f710 2024-03-25T18:17:44.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.312+0000 7fc2e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:44.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.312+0000 7fc2e2ffd640 1 -- 172.21.15.64:0/2811844174 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fc2d8001ba0 con 0x7fc2e413f710 2024-03-25T18:17:44.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.312+0000 7fc2e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.312+0000 7fc2e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:44.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.312+0000 7fc2e2ffd640 10 monclient: dump: 2024-03-25T18:17:44.314 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:44.314 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:44.314 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:44.314 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:44.314 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.314 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:44.314 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:44.314 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:44.314 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:44.315 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:44.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.312+0000 7fc2ea8d9640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:44.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.313+0000 7fc2ea8d9640 4 set_mon_vals no callback set 2024-03-25T18:17:44.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.313+0000 7fc2ea8d9640 10 monclient: shutdown 2024-03-25T18:17:44.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.313+0000 7fc2ea8d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:44.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.313+0000 7fc2ea8d9640 1 -- 172.21.15.64:0/2811844174 >> v1:172.21.15.64:6789/0 conn(0x7fc2e413f710 legacy=0x7fc2e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.314+0000 7fc2ea8d9640 1 -- 172.21.15.64:0/2811844174 shutdown_connections 2024-03-25T18:17:44.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.314+0000 7fc2ea8d9640 1 -- 172.21.15.64:0/2811844174 >> 172.21.15.64:0/2811844174 conn(0x7fc2e40b7ea0 msgr2=0x7fc2e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:44.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.314+0000 7fc2ea8d9640 1 -- 172.21.15.64:0/2811844174 shutdown_connections 2024-03-25T18:17:44.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.314+0000 7fc2ea8d9640 1 -- 172.21.15.64:0/2811844174 wait complete. 2024-03-25T18:17:44.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.315+0000 7fc2ea8d9640 10 monclient: build_initial_monmap 2024-03-25T18:17:44.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.315+0000 7fc2ea8d9640 10 monclient: monmap: 2024-03-25T18:17:44.316 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:17:44.316 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:44.317 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:17:44.317 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:17:44.317 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:44.317 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:17:44.317 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:17:44.317 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:17:44.317 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:17:44.317 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:44.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.315+0000 7fc2ea8d9640 1 librados: starting msgr at 2024-03-25T18:17:44.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.315+0000 7fc2ea8d9640 1 librados: starting objecter 2024-03-25T18:17:44.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.315+0000 7fc2ea8d9640 1 Processor -- start 2024-03-25T18:17:44.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.315+0000 7fc2ea8d9640 1 -- start start 2024-03-25T18:17:44.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.315+0000 7fc2ea8d9640 1 librados: setting wanted keys 2024-03-25T18:17:44.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.315+0000 7fc2ea8d9640 1 librados: calling monclient init 2024-03-25T18:17:44.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.315+0000 7fc2ea8d9640 10 monclient: init 2024-03-25T18:17:44.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.316+0000 7fc2ea8d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:44.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.316+0000 7fc2ea8d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:44.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.316+0000 7fc2ea8d9640 10 monclient(hunting): picked mon.noname-b con 0x7fc2e40a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:17:44.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.316+0000 7fc2ea8d9640 10 monclient(hunting): picked mon.noname-a con 0x7fc2e413f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:17:44.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.316+0000 7fc2ea8d9640 10 monclient(hunting): picked mon.noname-c con 0x7fc2e41430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:17:44.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.316+0000 7fc2ea8d9640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e41da880 con 0x7fc2e413f710 2024-03-25T18:17:44.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.316+0000 7fc2ea8d9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e41399a0 con 0x7fc2e40a98f0 2024-03-25T18:17:44.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.316+0000 7fc2ea8d9640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e41db700 con 0x7fc2e41430c0 2024-03-25T18:17:44.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.316+0000 7fc2ea8d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:44.319 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.316+0000 7fc2ea8d9640 10 monclient(hunting): authenticate will time out at 7450.555178s 2024-03-25T18:17:44.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.322+0000 7fc2e3fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc2e40a98f0 0x7fc2e41daff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:33360/0 (socket says 172.21.15.64:33360) 2024-03-25T18:17:44.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.322+0000 7fc2e3fff640 1 -- 172.21.15.64:0/3768926683 learned_addr learned my addr 172.21.15.64:0/3768926683 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:17:44.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.323+0000 7fc2e0ff9640 10 client.?.objecter ms_handle_connect 0x7fc2e40a98f0 2024-03-25T18:17:44.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.323+0000 7fc2e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:44.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.327+0000 7fc2e0ff9640 10 client.?.objecter ms_handle_connect 0x7fc2e413f710 2024-03-25T18:17:44.328 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.327+0000 7fc2e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:44.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.335+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761265679 0 0) 0x7fc2e41399a0 con 0x7fc2e40a98f0 2024-03-25T18:17:44.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.335+0000 7fc2e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.335+0000 7fc2e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.335+0000 7fc2e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.335+0000 7fc2e0ff9640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:17:44.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.335+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2d0004eb0 con 0x7fc2e40a98f0 2024-03-25T18:17:44.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.342+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993695536 0 0) 0x7fc2e41da880 con 0x7fc2e413f710 2024-03-25T18:17:44.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.342+0000 7fc2e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.342+0000 7fc2e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.342+0000 7fc2e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.342+0000 7fc2e0ff9640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:17:44.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.342+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2e41399a0 con 0x7fc2e413f710 2024-03-25T18:17:44.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.347+0000 7fc2e0ff9640 10 client.?.objecter ms_handle_connect 0x7fc2e41430c0 2024-03-25T18:17:44.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.347+0000 7fc2e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:44.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.355+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1781786784 0 0) 0x7fc2e41db700 con 0x7fc2e41430c0 2024-03-25T18:17:44.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.355+0000 7fc2e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.355+0000 7fc2e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.355+0000 7fc2e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.355+0000 7fc2e0ff9640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:17:44.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.355+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2e41da880 con 0x7fc2e41430c0 2024-03-25T18:17:44.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.360+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1394707411 0 0) 0x7fc2d0004eb0 con 0x7fc2e40a98f0 2024-03-25T18:17:44.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.360+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2e41db700 con 0x7fc2e40a98f0 2024-03-25T18:17:44.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.360+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fc2d4001a30 con 0x7fc2e40a98f0 2024-03-25T18:17:44.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.360+0000 7fc2e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.360+0000 7fc2e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:44.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.360+0000 7fc2e0ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:44.361 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:44.361 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:44.362 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:44.362 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:44.362 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.362 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:44.362 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:44.362 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:44.362 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:44.362 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:44.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.360+0000 7fc2e0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:44.367 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.366+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1109678698 0 0) 0x7fc2e41399a0 con 0x7fc2e413f710 2024-03-25T18:17:44.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.366+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2d0004eb0 con 0x7fc2e413f710 2024-03-25T18:17:44.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.366+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fc2d8001ae0 con 0x7fc2e413f710 2024-03-25T18:17:44.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.366+0000 7fc2e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.366+0000 7fc2e0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:44.368 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.366+0000 7fc2e0ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:44.368 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:44.368 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:44.368 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:44.368 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:44.368 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.368 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:44.368 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:44.368 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:44.368 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:44.369 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:44.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.367+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3597110044 0 0) 0x7fc2e41da880 con 0x7fc2e41430c0 2024-03-25T18:17:44.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.367+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2e41399a0 con 0x7fc2e41430c0 2024-03-25T18:17:44.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.367+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fc2dc0019e0 con 0x7fc2e41430c0 2024-03-25T18:17:44.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.367+0000 7fc2e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.367+0000 7fc2e0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:44.369 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.367+0000 7fc2e0ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:44.369 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:44.369 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:44.369 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:44.369 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:44.369 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.369 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:44.369 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:44.370 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:44.370 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:44.370 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:44.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 58121816 0 0) 0x7fc2e41db700 con 0x7fc2e40a98f0 2024-03-25T18:17:44.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 >> v1:172.21.15.64:6790/0 conn(0x7fc2e41430c0 legacy=0x7fc2e41e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 >> v1:172.21.15.64:6789/0 conn(0x7fc2e413f710 legacy=0x7fc2e4138410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2e0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:17:44.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2e0ff9640 1 monclient: found mon.b 2024-03-25T18:17:44.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:44.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2e0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:17:44.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e41daad0 con 0x7fc2e40a98f0 2024-03-25T18:17:44.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2e0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:17:44.372 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2e0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:17:44.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:44.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2ea8d9640 5 monclient: authenticate success, global_id 4594 2024-03-25T18:17:44.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2ea8d9640 10 monclient: _renew_subs 2024-03-25T18:17:44.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2ea8d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:17:44.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2ea8d9640 1 -- 172.21.15.64:0/3768926683 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e4138b20 con 0x7fc2e40a98f0 2024-03-25T18:17:44.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2ea8d9640 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:44.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2ea8d9640 10 monclient: _renew_subs 2024-03-25T18:17:44.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2ea8d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:17:44.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2ea8d9640 1 -- 172.21.15.64:0/3768926683 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e4137a70 con 0x7fc2e40a98f0 2024-03-25T18:17:44.373 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.370+0000 7fc2ea8d9640 1 librados: init done 2024-03-25T18:17:44.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.371+0000 7fc2ea8d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:44.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.371+0000 7fc2ea8d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:44.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.371+0000 7fc2ea8d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:17:44.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.371+0000 7fc2ea8d9640 1 -- 172.21.15.64:0/3768926683 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e40627b0 con 0x7fc2e40a98f0 2024-03-25T18:17:44.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.646+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2d4001c70 con 0x7fc2e40a98f0 2024-03-25T18:17:44.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.646+0000 7fc2e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:44.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.646+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fc2d4003a60 con 0x7fc2e40a98f0 2024-03-25T18:17:44.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.646+0000 7fc2e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.646+0000 7fc2e0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:44.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.646+0000 7fc2e0ff9640 10 monclient: dump: 2024-03-25T18:17:44.647 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:44.647 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:44.647 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:44.647 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:44.648 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.648 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:44.648 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:44.648 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:44.648 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:44.648 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:44.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.646+0000 7fc2e27fc640 4 set_mon_vals no callback set 2024-03-25T18:17:44.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.646+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fc2d4003c40 con 0x7fc2e40a98f0 2024-03-25T18:17:44.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.647+0000 7fc2e37fe640 10 monclient: get_auth_request con 0x7fc2d006e0b0 auth_method 0 2024-03-25T18:17:44.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.647+0000 7fc2e0ff9640 10 client.4594.objecter ms_handle_connect 0x7fc2d006e0b0 2024-03-25T18:17:44.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.651+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fc2d4054940 con 0x7fc2e40a98f0 2024-03-25T18:17:44.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.651+0000 7fc2e0ff9640 10 client.4594.objecter ms_dispatch 0x7fc2e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:44.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.651+0000 7fc2e0ff9640 3 client.4594.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:44.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.651+0000 7fc2e0ff9640 3 client.4594.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:44.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.651+0000 7fc2e0ff9640 20 client.4594.objecter dump_active .. 0 homeless 2024-03-25T18:17:44.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.651+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2d4050450 con 0x7fc2e40a98f0 2024-03-25T18:17:44.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.651+0000 7fc2e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:44.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.651+0000 7fc2e0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:44.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.915+0000 7fc2ea8d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:44.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.915+0000 7fc2ea8d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:44.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.915+0000 7fc2ea8d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:17:44.916 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.915+0000 7fc2ea8d9640 1 -- 172.21.15.64:0/3768926683 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2e4069f40 con 0x7fc2e40a98f0 2024-03-25T18:17:44.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.949+0000 7fc2e0ff9640 1 -- 172.21.15.64:0/3768926683 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fc2d4052380 con 0x7fc2e40a98f0 2024-03-25T18:17:44.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.950+0000 7fc2e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:44.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.950+0000 7fc2e0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:44.951 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:17:44.951 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:44.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.952+0000 7fc2ea8d9640 10 librados: watch_flush enter 2024-03-25T18:17:44.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.952+0000 7fc2ea8d9640 10 librados: watch_flush exit 2024-03-25T18:17:44.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.952+0000 7fc2ea8d9640 20 client.4594.objecter shutdown clearing up homeless session... 2024-03-25T18:17:44.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.952+0000 7fc2ea8d9640 10 client.4594.objecter successfully canceled tick 2024-03-25T18:17:44.953 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.952+0000 7fc2ea8d9640 1 -- 172.21.15.64:0/3768926683 >> v1:172.21.15.64:6800/2915543638 conn(0x7fc2d006e0b0 legacy=0x7fc2d0070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.952+0000 7fc2ea8d9640 10 monclient: shutdown 2024-03-25T18:17:44.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.952+0000 7fc2ea8d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:44.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.952+0000 7fc2ea8d9640 1 -- 172.21.15.64:0/3768926683 >> v1:172.21.15.86:6789/0 conn(0x7fc2e40a98f0 legacy=0x7fc2e41daff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.952+0000 7fc2ea8d9640 1 -- 172.21.15.64:0/3768926683 shutdown_connections 2024-03-25T18:17:44.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.952+0000 7fc2ea8d9640 1 -- 172.21.15.64:0/3768926683 >> 172.21.15.64:0/3768926683 conn(0x7fc2e40b7ea0 msgr2=0x7fc2e413d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:44.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.952+0000 7fc2ea8d9640 1 -- 172.21.15.64:0/3768926683 shutdown_connections 2024-03-25T18:17:44.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.952+0000 7fc2ea8d9640 1 -- 172.21.15.64:0/3768926683 wait complete. 2024-03-25T18:17:44.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:44.952+0000 7fc2ea8d9640 1 librados: shutdown 2024-03-25T18:17:44.966 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:51.773 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:51.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.887+0000 7f4fcb840640 10 monclient: get_monmap_and_config 2024-03-25T18:17:51.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.887+0000 7f4fcb840640 10 monclient: build_initial_monmap 2024-03-25T18:17:51.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.887+0000 7f4fcb840640 10 monclient: monmap: 2024-03-25T18:17:51.888 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:17:51.889 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:51.889 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:17:51.888577+0000 2024-03-25T18:17:51.889 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:17:51.888577+0000 2024-03-25T18:17:51.889 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:51.889 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:51.889 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:17:51.889 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:17:51.889 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:17:51.889 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:51.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.888+0000 7f4fcb840640 1 Processor -- start 2024-03-25T18:17:51.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.888+0000 7f4fcb840640 1 -- start start 2024-03-25T18:17:51.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.888+0000 7f4fcb840640 10 monclient: init 2024-03-25T18:17:51.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.889+0000 7f4fcb840640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:51.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.889+0000 7f4fcb840640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:51.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.889+0000 7f4fcb840640 10 monclient(hunting): picked mon.noname-c con 0x7f4fc40a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:17:51.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.889+0000 7f4fcb840640 10 monclient(hunting): picked mon.noname-a con 0x7f4fc413f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:17:51.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.889+0000 7f4fcb840640 10 monclient(hunting): picked mon.noname-b con 0x7f4fc41430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:17:51.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.889+0000 7f4fcb840640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc4142520 con 0x7f4fc413f710 2024-03-25T18:17:51.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.889+0000 7f4fcb840640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc40b28a0 con 0x7f4fc40a98f0 2024-03-25T18:17:51.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.889+0000 7f4fcb840640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc40b2a30 con 0x7f4fc41430c0 2024-03-25T18:17:51.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.889+0000 7f4fcb840640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:51.891 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.889+0000 7f4fcb840640 10 monclient(hunting): authenticate will time out at 7458.127885s 2024-03-25T18:17:51.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.903+0000 7f4fc95dc640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4fc40a98f0 0x7f4fc40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:58764/0 (socket says 172.21.15.64:58764) 2024-03-25T18:17:51.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.903+0000 7f4fc95dc640 1 -- 172.21.15.64:0/727371131 learned_addr learned my addr 172.21.15.64:0/727371131 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:17:51.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.912+0000 7f4fbbfff640 1 -- 172.21.15.64:0/727371131 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782691881 0 0) 0x7f4fc40b28a0 con 0x7f4fc40a98f0 2024-03-25T18:17:51.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.912+0000 7f4fbbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:51.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.912+0000 7f4fbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:51.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.912+0000 7f4fbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:51.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.912+0000 7f4fbbfff640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:17:51.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.912+0000 7f4fbbfff640 1 -- 172.21.15.64:0/727371131 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fb0003520 con 0x7f4fc40a98f0 2024-03-25T18:17:51.918 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.918+0000 7f4fbbfff640 1 -- 172.21.15.64:0/727371131 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557390179 0 0) 0x7f4fc40b2a30 con 0x7f4fc41430c0 2024-03-25T18:17:51.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.918+0000 7f4fbbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:51.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.918+0000 7f4fbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:51.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.918+0000 7f4fbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:51.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.918+0000 7f4fbbfff640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:17:51.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.918+0000 7f4fbbfff640 1 -- 172.21.15.64:0/727371131 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fc40b28a0 con 0x7f4fc41430c0 2024-03-25T18:17:51.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.923+0000 7f4fbbfff640 1 -- 172.21.15.64:0/727371131 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 511070534 0 0) 0x7f4fb0003520 con 0x7f4fc40a98f0 2024-03-25T18:17:51.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.923+0000 7f4fbbfff640 1 -- 172.21.15.64:0/727371131 >> v1:172.21.15.64:6790/0 conn(0x7f4fc41430c0 legacy=0x7f4fc414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:51.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.923+0000 7f4fbbfff640 1 -- 172.21.15.64:0/727371131 >> v1:172.21.15.64:6789/0 conn(0x7f4fc413f710 legacy=0x7f4fc41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:17:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.923+0000 7f4fbbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.923+0000 7f4fbbfff640 1 monclient: found mon.noname-c 2024-03-25T18:17:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.923+0000 7f4fbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.923+0000 7f4fbbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:17:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.923+0000 7f4fbbfff640 1 -- 172.21.15.64:0/727371131 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc40b44c0 con 0x7f4fc40a98f0 2024-03-25T18:17:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.923+0000 7f4fbbfff640 10 monclient: _finish_auth 0 2024-03-25T18:17:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.923+0000 7f4fbbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.923+0000 7f4fbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.923+0000 7f4fbbfff640 1 -- 172.21.15.64:0/727371131 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4fc0001690 con 0x7f4fc40a98f0 2024-03-25T18:17:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.923+0000 7f4fbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.923+0000 7f4fbbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:51.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.923+0000 7f4fbbfff640 10 monclient: dump: 2024-03-25T18:17:51.925 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:51.925 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:51.926 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:51.926 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:51.926 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:51.926 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:51.926 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:51.926 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:51.926 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:51.926 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:51.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.923+0000 7f4fcb840640 5 monclient: authenticate success, global_id 4597 2024-03-25T18:17:51.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.923+0000 7f4fcb840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:51.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.938+0000 7f4fbbfff640 1 -- 172.21.15.64:0/727371131 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fc00017f0 con 0x7f4fc40a98f0 2024-03-25T18:17:51.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.938+0000 7f4fbbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:51.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.938+0000 7f4fbbfff640 1 -- 172.21.15.64:0/727371131 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4fc0001c00 con 0x7f4fc40a98f0 2024-03-25T18:17:51.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.938+0000 7f4fcb840640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:51.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.938+0000 7f4fcb840640 4 set_mon_vals no callback set 2024-03-25T18:17:51.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.938+0000 7f4fbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:51.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.938+0000 7f4fbbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:51.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.938+0000 7f4fbbfff640 10 monclient: dump: 2024-03-25T18:17:51.939 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:51.939 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:51.940 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:51.940 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:51.940 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:51.940 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:51.940 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:51.940 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:51.940 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:51.940 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:51.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.938+0000 7f4fcb840640 10 monclient: shutdown 2024-03-25T18:17:51.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.938+0000 7f4fcb840640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:51.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.939+0000 7f4fcb840640 1 -- 172.21.15.64:0/727371131 >> v1:172.21.15.86:6789/0 conn(0x7f4fc40a98f0 legacy=0x7f4fc40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:51.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.939+0000 7f4fcb840640 1 -- 172.21.15.64:0/727371131 shutdown_connections 2024-03-25T18:17:51.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.939+0000 7f4fcb840640 1 -- 172.21.15.64:0/727371131 >> 172.21.15.64:0/727371131 conn(0x7f4fc40b7ea0 msgr2=0x7f4fc40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:51.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.939+0000 7f4fcb840640 1 -- 172.21.15.64:0/727371131 shutdown_connections 2024-03-25T18:17:51.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.939+0000 7f4fcb840640 1 -- 172.21.15.64:0/727371131 wait complete. 2024-03-25T18:17:51.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.940+0000 7f4fcb840640 10 monclient: build_initial_monmap 2024-03-25T18:17:51.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.940+0000 7f4fcb840640 10 monclient: monmap: 2024-03-25T18:17:51.942 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:17:51.942 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:51.942 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:17:51.942 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:17:51.942 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:51.942 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:17:51.942 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:17:51.942 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:17:51.943 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:17:51.943 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:51.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.940+0000 7f4fcb840640 1 librados: starting msgr at 2024-03-25T18:17:51.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.940+0000 7f4fcb840640 1 librados: starting objecter 2024-03-25T18:17:51.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.940+0000 7f4fcb840640 1 Processor -- start 2024-03-25T18:17:51.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.940+0000 7f4fcb840640 1 -- start start 2024-03-25T18:17:51.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.940+0000 7f4fcb840640 1 librados: setting wanted keys 2024-03-25T18:17:51.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.940+0000 7f4fcb840640 1 librados: calling monclient init 2024-03-25T18:17:51.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.940+0000 7f4fcb840640 10 monclient: init 2024-03-25T18:17:51.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.941+0000 7f4fcb840640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:51.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.941+0000 7f4fcb840640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:51.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.941+0000 7f4fcb840640 10 monclient(hunting): picked mon.noname-a con 0x7f4fc40a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:17:51.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.941+0000 7f4fcb840640 10 monclient(hunting): picked mon.noname-c con 0x7f4fc413f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:17:51.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.941+0000 7f4fcb840640 10 monclient(hunting): picked mon.noname-b con 0x7f4fc41430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:17:51.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.941+0000 7f4fcb840640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc41d41e0 con 0x7f4fc40a98f0 2024-03-25T18:17:51.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.941+0000 7f4fcb840640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc41d4430 con 0x7f4fc41430c0 2024-03-25T18:17:51.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.941+0000 7f4fcb840640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc4138cf0 con 0x7f4fc413f710 2024-03-25T18:17:51.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.941+0000 7f4fcb840640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:51.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.941+0000 7f4fcb840640 10 monclient(hunting): authenticate will time out at 7458.180119s 2024-03-25T18:17:51.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.948+0000 7f4fc95dc640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f4fc40a98f0 0x7f4fc41385e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:52570/0 (socket says 172.21.15.64:52570) 2024-03-25T18:17:51.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.948+0000 7f4fc95dc640 1 -- 172.21.15.64:0/4282453531 learned_addr learned my addr 172.21.15.64:0/4282453531 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:17:51.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.949+0000 7f4fb9ffb640 10 client.?.objecter ms_handle_connect 0x7f4fc40a98f0 2024-03-25T18:17:51.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.949+0000 7f4fb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:51.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.950+0000 7f4fb9ffb640 10 client.?.objecter ms_handle_connect 0x7f4fc41430c0 2024-03-25T18:17:51.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:51.950+0000 7f4fb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:52.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.008+0000 7f4fb9ffb640 10 client.?.objecter ms_handle_connect 0x7f4fc413f710 2024-03-25T18:17:52.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.008+0000 7f4fb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:52.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.034+0000 7f4fb9ffb640 1 -- 172.21.15.64:0/4282453531 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960438008 0 0) 0x7f4fc4138cf0 con 0x7f4fc413f710 2024-03-25T18:17:52.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.034+0000 7f4fb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:52.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.034+0000 7f4fb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:52.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.034+0000 7f4fb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:52.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.034+0000 7f4fb9ffb640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:17:52.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.034+0000 7f4fb9ffb640 1 -- 172.21.15.64:0/4282453531 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fa8004eb0 con 0x7f4fc413f710 2024-03-25T18:17:52.055 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.054+0000 7f4fb9ffb640 1 -- 172.21.15.64:0/4282453531 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4144066810 0 0) 0x7f4fa8004eb0 con 0x7f4fc413f710 2024-03-25T18:17:52.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.055+0000 7f4fb9ffb640 1 -- 172.21.15.64:0/4282453531 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fc4138cf0 con 0x7f4fc413f710 2024-03-25T18:17:52.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.055+0000 7f4fb9ffb640 1 -- 172.21.15.64:0/4282453531 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4fb4001510 con 0x7f4fc413f710 2024-03-25T18:17:52.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.055+0000 7f4fb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:52.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.055+0000 7f4fb9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:52.056 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.055+0000 7f4fb9ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:52.056 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:52.056 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:52.056 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:52.056 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:52.056 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:52.056 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:52.056 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:52.056 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:52.056 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:52.057 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:52.057 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.055+0000 7f4fb9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:52.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.062+0000 7f4fb9ffb640 1 -- 172.21.15.64:0/4282453531 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988391825 0 0) 0x7f4fc41d4430 con 0x7f4fc41430c0 2024-03-25T18:17:52.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.062+0000 7f4fb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:52.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.062+0000 7f4fb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:52.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.062+0000 7f4fb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:52.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.062+0000 7f4fb9ffb640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:17:52.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.062+0000 7f4fb9ffb640 1 -- 172.21.15.64:0/4282453531 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fa8004eb0 con 0x7f4fc41430c0 2024-03-25T18:17:52.066 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fb9ffb640 1 -- 172.21.15.64:0/4282453531 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3994404963 0 0) 0x7f4fc4138cf0 con 0x7f4fc413f710 2024-03-25T18:17:52.066 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fb9ffb640 1 -- 172.21.15.64:0/4282453531 >> v1:172.21.15.86:6789/0 conn(0x7f4fc41430c0 legacy=0x7f4fc41e1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:52.066 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fb9ffb640 1 -- 172.21.15.64:0/4282453531 >> v1:172.21.15.64:6789/0 conn(0x7f4fc40a98f0 legacy=0x7f4fc41385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:52.066 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fb9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:52.066 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fb9ffb640 1 monclient: found mon.c 2024-03-25T18:17:52.066 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fb9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:52.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fb9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:17:52.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fb9ffb640 1 -- 172.21.15.64:0/4282453531 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc4138e80 con 0x7f4fc413f710 2024-03-25T18:17:52.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fb9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:52.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fb9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:52.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:52.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fcb840640 5 monclient: authenticate success, global_id 4598 2024-03-25T18:17:52.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fcb840640 10 monclient: _renew_subs 2024-03-25T18:17:52.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fcb840640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:17:52.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fcb840640 1 -- 172.21.15.64:0/4282453531 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc41d5120 con 0x7f4fc413f710 2024-03-25T18:17:52.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fcb840640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:52.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fcb840640 10 monclient: _renew_subs 2024-03-25T18:17:52.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fcb840640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:17:52.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fcb840640 1 -- 172.21.15.64:0/4282453531 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc41350f0 con 0x7f4fc413f710 2024-03-25T18:17:52.067 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.065+0000 7f4fcb840640 1 librados: init done 2024-03-25T18:17:52.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.066+0000 7f4fcb840640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:52.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.066+0000 7f4fcb840640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:52.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.066+0000 7f4fcb840640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:17:52.068 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.066+0000 7f4fcb840640 1 -- 172.21.15.64:0/4282453531 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc40b3ef0 con 0x7f4fc413f710 2024-03-25T18:17:52.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.388+0000 7f4fb9ffb640 1 -- 172.21.15.64:0/4282453531 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fb4001a60 con 0x7f4fc413f710 2024-03-25T18:17:52.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.388+0000 7f4fb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:52.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.388+0000 7f4fb9ffb640 1 -- 172.21.15.64:0/4282453531 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f4fb4003910 con 0x7f4fc413f710 2024-03-25T18:17:52.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.388+0000 7f4fb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:52.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.388+0000 7f4fb9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:52.389 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.388+0000 7f4fb9ffb640 10 monclient: dump: 2024-03-25T18:17:52.390 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:52.390 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:52.390 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:52.390 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:52.390 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:52.390 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:52.390 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:52.390 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:52.390 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:52.390 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:52.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.388+0000 7f4fbb7fe640 4 set_mon_vals no callback set 2024-03-25T18:17:52.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.388+0000 7f4fb9ffb640 1 -- 172.21.15.64:0/4282453531 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f4fb40198e0 con 0x7f4fc413f710 2024-03-25T18:17:52.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.389+0000 7f4fc95dc640 10 monclient: get_auth_request con 0x7f4fa806dd90 auth_method 0 2024-03-25T18:17:52.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.389+0000 7f4fb9ffb640 10 client.4598.objecter ms_handle_connect 0x7f4fa806dd90 2024-03-25T18:17:52.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.394+0000 7f4fb9ffb640 1 -- 172.21.15.64:0/4282453531 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f4fb4053520 con 0x7f4fc413f710 2024-03-25T18:17:52.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.394+0000 7f4fb9ffb640 10 client.4598.objecter ms_dispatch 0x7f4fc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:52.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.394+0000 7f4fb9ffb640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:52.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.394+0000 7f4fb9ffb640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:52.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.394+0000 7f4fb9ffb640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-25T18:17:52.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.394+0000 7f4fb9ffb640 1 -- 172.21.15.64:0/4282453531 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4fb404ee60 con 0x7f4fc413f710 2024-03-25T18:17:52.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.394+0000 7f4fb9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:52.395 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.394+0000 7f4fb9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:52.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.643+0000 7f4fcb840640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:52.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.643+0000 7f4fcb840640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:52.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.643+0000 7f4fcb840640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:17:52.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.643+0000 7f4fcb840640 1 -- 172.21.15.64:0/4282453531 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fc41d45c0 con 0x7f4fc413f710 2024-03-25T18:17:52.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.677+0000 7f4fb9ffb640 1 -- 172.21.15.64:0/4282453531 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f4fb4050f60 con 0x7f4fc413f710 2024-03-25T18:17:52.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.677+0000 7f4fb9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:52.678 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.677+0000 7f4fb9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:52.678 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:17:52.678 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:52.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.679+0000 7f4fcb840640 10 librados: watch_flush enter 2024-03-25T18:17:52.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.679+0000 7f4fcb840640 10 librados: watch_flush exit 2024-03-25T18:17:52.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.679+0000 7f4fcb840640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-25T18:17:52.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.679+0000 7f4fcb840640 10 client.4598.objecter successfully canceled tick 2024-03-25T18:17:52.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.679+0000 7f4fcb840640 1 -- 172.21.15.64:0/4282453531 >> v1:172.21.15.64:6800/2915543638 conn(0x7f4fa806dd90 legacy=0x7f4fa8070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:52.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.679+0000 7f4fcb840640 10 monclient: shutdown 2024-03-25T18:17:52.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.679+0000 7f4fcb840640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:52.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.679+0000 7f4fcb840640 1 -- 172.21.15.64:0/4282453531 >> v1:172.21.15.64:6790/0 conn(0x7f4fc413f710 legacy=0x7f4fc41d4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:52.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.680+0000 7f4fcb840640 1 -- 172.21.15.64:0/4282453531 shutdown_connections 2024-03-25T18:17:52.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.680+0000 7f4fcb840640 1 -- 172.21.15.64:0/4282453531 >> 172.21.15.64:0/4282453531 conn(0x7f4fc40b7ea0 msgr2=0x7f4fc4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:52.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.680+0000 7f4fcb840640 1 -- 172.21.15.64:0/4282453531 shutdown_connections 2024-03-25T18:17:52.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.680+0000 7f4fcb840640 1 -- 172.21.15.64:0/4282453531 wait complete. 2024-03-25T18:17:52.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:52.680+0000 7f4fcb840640 1 librados: shutdown 2024-03-25T18:17:52.692 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:59.498 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:59.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.611+0000 7f07357af640 10 monclient: get_monmap_and_config 2024-03-25T18:17:59.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.611+0000 7f07357af640 10 monclient: build_initial_monmap 2024-03-25T18:17:59.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.611+0000 7f07357af640 10 monclient: monmap: 2024-03-25T18:17:59.612 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:17:59.612 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:59.613 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:17:59.612338+0000 2024-03-25T18:17:59.613 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:17:59.612338+0000 2024-03-25T18:17:59.613 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:59.613 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:59.613 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:17:59.613 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:17:59.613 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:17:59.613 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:59.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.612+0000 7f07357af640 1 Processor -- start 2024-03-25T18:17:59.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.612+0000 7f07357af640 1 -- start start 2024-03-25T18:17:59.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.612+0000 7f07357af640 10 monclient: init 2024-03-25T18:17:59.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.613+0000 7f07357af640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:59.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.613+0000 7f07357af640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:59.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.613+0000 7f07357af640 10 monclient(hunting): picked mon.noname-c con 0x7f073006ab80 addr v1:172.21.15.86:6789/0 2024-03-25T18:17:59.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.613+0000 7f07357af640 10 monclient(hunting): picked mon.noname-a con 0x7f073006af60 addr v1:172.21.15.64:6789/0 2024-03-25T18:17:59.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.613+0000 7f07357af640 10 monclient(hunting): picked mon.noname-b con 0x7f073013f990 addr v1:172.21.15.64:6790/0 2024-03-25T18:17:59.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.613+0000 7f07357af640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07300afa60 con 0x7f073006af60 2024-03-25T18:17:59.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.613+0000 7f07357af640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07300afce0 con 0x7f073006ab80 2024-03-25T18:17:59.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.613+0000 7f07357af640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07300b1150 con 0x7f073013f990 2024-03-25T18:17:59.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.613+0000 7f07357af640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:59.615 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.613+0000 7f07357af640 10 monclient(hunting): authenticate will time out at 7465.851694s 2024-03-25T18:17:59.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.621+0000 7f072e7fc640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f073006af60 0x7f07301458c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:36066/0 (socket says 172.21.15.64:36066) 2024-03-25T18:17:59.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.621+0000 7f072e7fc640 1 -- 172.21.15.64:0/357681399 learned_addr learned my addr 172.21.15.64:0/357681399 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:17:59.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.636+0000 7f072dffb640 1 -- 172.21.15.64:0/357681399 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621095614 0 0) 0x7f07300afa60 con 0x7f073006af60 2024-03-25T18:17:59.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.636+0000 7f072dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:59.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.636+0000 7f072dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:59.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.636+0000 7f072dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:59.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.636+0000 7f072dffb640 10 monclient(hunting): my global_id is 4599 2024-03-25T18:17:59.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.636+0000 7f072dffb640 1 -- 172.21.15.64:0/357681399 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0718003520 con 0x7f073006af60 2024-03-25T18:17:59.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.645+0000 7f072dffb640 1 -- 172.21.15.64:0/357681399 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299192507 0 0) 0x7f07300afce0 con 0x7f073006ab80 2024-03-25T18:17:59.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.645+0000 7f072dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:59.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.645+0000 7f072dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:59.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.645+0000 7f072dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:59.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.645+0000 7f072dffb640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:17:59.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.645+0000 7f072dffb640 1 -- 172.21.15.64:0/357681399 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07300afa60 con 0x7f073006ab80 2024-03-25T18:17:59.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.648+0000 7f072dffb640 1 -- 172.21.15.64:0/357681399 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272116927 0 0) 0x7f07300b1150 con 0x7f073013f990 2024-03-25T18:17:59.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.648+0000 7f072dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:59.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.648+0000 7f072dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:59.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.648+0000 7f072dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:59.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.648+0000 7f072dffb640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:17:59.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.648+0000 7f072dffb640 1 -- 172.21.15.64:0/357681399 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07300afce0 con 0x7f073013f990 2024-03-25T18:17:59.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.653+0000 7f072dffb640 1 -- 172.21.15.64:0/357681399 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 971687028 0 0) 0x7f0718003520 con 0x7f073006af60 2024-03-25T18:17:59.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.653+0000 7f072dffb640 1 -- 172.21.15.64:0/357681399 >> v1:172.21.15.64:6790/0 conn(0x7f073013f990 legacy=0x7f073014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.653+0000 7f072dffb640 1 -- 172.21.15.64:0/357681399 >> v1:172.21.15.86:6789/0 conn(0x7f073006ab80 legacy=0x7f07300724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.653+0000 7f072dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:59.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.653+0000 7f072dffb640 1 monclient: found mon.noname-a 2024-03-25T18:17:59.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.653+0000 7f072dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:59.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.653+0000 7f072dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:17:59.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.653+0000 7f072dffb640 1 -- 172.21.15.64:0/357681399 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07300aff20 con 0x7f073006af60 2024-03-25T18:17:59.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.653+0000 7f072dffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:59.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.653+0000 7f072dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:59.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.653+0000 7f072dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:59.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.653+0000 7f072dffb640 1 -- 172.21.15.64:0/357681399 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0724001690 con 0x7f073006af60 2024-03-25T18:17:59.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.653+0000 7f07357af640 5 monclient: authenticate success, global_id 4599 2024-03-25T18:17:59.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.653+0000 7f07357af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:59.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.653+0000 7f072dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:59.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.653+0000 7f072dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:59.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.653+0000 7f072dffb640 10 monclient: dump: 2024-03-25T18:17:59.655 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:59.655 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:59.655 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:59.655 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:59.656 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:59.656 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:59.656 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:59.656 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:59.656 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:59.656 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:59.656 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.653+0000 7f07357af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:59.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.670+0000 7f072dffb640 1 -- 172.21.15.64:0/357681399 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07240017f0 con 0x7f073006af60 2024-03-25T18:17:59.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.670+0000 7f072dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:59.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.670+0000 7f072dffb640 1 -- 172.21.15.64:0/357681399 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0724001c00 con 0x7f073006af60 2024-03-25T18:17:59.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.670+0000 7f072dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:59.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.671+0000 7f072dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:59.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.671+0000 7f072dffb640 10 monclient: dump: 2024-03-25T18:17:59.672 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:59.672 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:59.672 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:59.672 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:59.672 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:59.672 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:59.672 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:59.672 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:59.673 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:59.673 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:59.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.671+0000 7f07357af640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:59.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.671+0000 7f07357af640 4 set_mon_vals no callback set 2024-03-25T18:17:59.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.671+0000 7f07357af640 10 monclient: shutdown 2024-03-25T18:17:59.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.671+0000 7f07357af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:59.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.671+0000 7f07357af640 1 -- 172.21.15.64:0/357681399 >> v1:172.21.15.64:6789/0 conn(0x7f073006af60 legacy=0x7f07301458c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.671+0000 7f07357af640 1 -- 172.21.15.64:0/357681399 shutdown_connections 2024-03-25T18:17:59.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.671+0000 7f07357af640 1 -- 172.21.15.64:0/357681399 >> 172.21.15.64:0/357681399 conn(0x7f073006c830 msgr2=0x7f07300b4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:59.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.671+0000 7f07357af640 1 -- 172.21.15.64:0/357681399 shutdown_connections 2024-03-25T18:17:59.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.671+0000 7f07357af640 1 -- 172.21.15.64:0/357681399 wait complete. 2024-03-25T18:17:59.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.672+0000 7f07357af640 10 monclient: build_initial_monmap 2024-03-25T18:17:59.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.672+0000 7f07357af640 10 monclient: monmap: 2024-03-25T18:17:59.674 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:17:59.675 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:59.675 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:17:59.675 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:17:59.675 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:59.675 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:17:59.675 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:17:59.675 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:17:59.675 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:17:59.675 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:59.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.673+0000 7f07357af640 1 librados: starting msgr at 2024-03-25T18:17:59.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.673+0000 7f07357af640 1 librados: starting objecter 2024-03-25T18:17:59.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.673+0000 7f07357af640 1 Processor -- start 2024-03-25T18:17:59.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.673+0000 7f07357af640 1 -- start start 2024-03-25T18:17:59.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.673+0000 7f07357af640 1 librados: setting wanted keys 2024-03-25T18:17:59.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.673+0000 7f07357af640 1 librados: calling monclient init 2024-03-25T18:17:59.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.673+0000 7f07357af640 10 monclient: init 2024-03-25T18:17:59.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.674+0000 7f07357af640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:59.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.674+0000 7f07357af640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:59.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.674+0000 7f07357af640 10 monclient(hunting): picked mon.noname-a con 0x7f073006ab80 addr v1:172.21.15.64:6789/0 2024-03-25T18:17:59.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.674+0000 7f07357af640 10 monclient(hunting): picked mon.noname-b con 0x7f073013f990 addr v1:172.21.15.86:6789/0 2024-03-25T18:17:59.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.674+0000 7f07357af640 10 monclient(hunting): picked mon.noname-c con 0x7f0730145050 addr v1:172.21.15.64:6790/0 2024-03-25T18:17:59.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.674+0000 7f07357af640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07301358f0 con 0x7f073006ab80 2024-03-25T18:17:59.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.674+0000 7f07357af640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0730135b60 con 0x7f073013f990 2024-03-25T18:17:59.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.674+0000 7f07357af640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0730140a20 con 0x7f0730145050 2024-03-25T18:17:59.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.674+0000 7f07357af640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:59.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.674+0000 7f07357af640 10 monclient(hunting): authenticate will time out at 7465.912808s 2024-03-25T18:17:59.691 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.691+0000 7f072f7fe640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f0730145050 0x7f07301362d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:35284/0 (socket says 172.21.15.64:35284) 2024-03-25T18:17:59.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.691+0000 7f072f7fe640 1 -- 172.21.15.64:0/3103818212 learned_addr learned my addr 172.21.15.64:0/3103818212 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:17:59.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.691+0000 7f07177fe640 10 client.?.objecter ms_handle_connect 0x7f0730145050 2024-03-25T18:17:59.692 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.691+0000 7f07177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:59.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.693+0000 7f07177fe640 10 client.?.objecter ms_handle_connect 0x7f073006ab80 2024-03-25T18:17:59.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.693+0000 7f07177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:59.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.700+0000 7f07177fe640 1 -- 172.21.15.64:0/3103818212 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521894210 0 0) 0x7f0730140a20 con 0x7f0730145050 2024-03-25T18:17:59.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.700+0000 7f07177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:59.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.700+0000 7f07177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:59.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.700+0000 7f07177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:59.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.700+0000 7f07177fe640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:17:59.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.700+0000 7f07177fe640 1 -- 172.21.15.64:0/3103818212 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f071c004eb0 con 0x7f0730145050 2024-03-25T18:17:59.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.701+0000 7f07177fe640 10 client.?.objecter ms_handle_connect 0x7f073013f990 2024-03-25T18:17:59.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.701+0000 7f07177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:59.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.702+0000 7f07177fe640 1 -- 172.21.15.64:0/3103818212 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741996897 0 0) 0x7f07301358f0 con 0x7f073006ab80 2024-03-25T18:17:59.702 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.702+0000 7f07177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:59.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.702+0000 7f07177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:59.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.702+0000 7f07177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:59.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.702+0000 7f07177fe640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:17:59.703 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.702+0000 7f07177fe640 1 -- 172.21.15.64:0/3103818212 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0730140a20 con 0x7f073006ab80 2024-03-25T18:17:59.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.709+0000 7f07177fe640 1 -- 172.21.15.64:0/3103818212 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1665783208 0 0) 0x7f071c004eb0 con 0x7f0730145050 2024-03-25T18:17:59.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.710+0000 7f07177fe640 1 -- 172.21.15.64:0/3103818212 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07301358f0 con 0x7f0730145050 2024-03-25T18:17:59.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.710+0000 7f07177fe640 1 -- 172.21.15.64:0/3103818212 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0728001cd0 con 0x7f0730145050 2024-03-25T18:17:59.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.710+0000 7f07177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:59.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.710+0000 7f07177fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:59.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.710+0000 7f07177fe640 10 monclient(hunting): dump: 2024-03-25T18:17:59.711 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:17:59.711 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:17:59.711 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:59.711 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:17:59.711 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:59.711 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:17:59.711 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:17:59.711 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:17:59.712 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:17:59.712 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:17:59.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.710+0000 7f07177fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:59.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.720+0000 7f07177fe640 1 -- 172.21.15.64:0/3103818212 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1642499608 0 0) 0x7f07301358f0 con 0x7f0730145050 2024-03-25T18:17:59.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.720+0000 7f07177fe640 1 -- 172.21.15.64:0/3103818212 >> v1:172.21.15.86:6789/0 conn(0x7f073013f990 legacy=0x7f07301401c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.720+0000 7f07177fe640 1 -- 172.21.15.64:0/3103818212 >> v1:172.21.15.64:6789/0 conn(0x7f073006ab80 legacy=0x7f0730140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.720+0000 7f07177fe640 10 monclient: _finish_hunting 0 2024-03-25T18:17:59.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.720+0000 7f07177fe640 1 monclient: found mon.c 2024-03-25T18:17:59.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.720+0000 7f07177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:59.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.720+0000 7f07177fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:17:59.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.720+0000 7f07177fe640 1 -- 172.21.15.64:0/3103818212 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07301e66c0 con 0x7f0730145050 2024-03-25T18:17:59.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.721+0000 7f07177fe640 10 monclient: _finish_auth 0 2024-03-25T18:17:59.722 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.721+0000 7f07177fe640 10 monclient: _check_auth_tickets 2024-03-25T18:17:59.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.721+0000 7f07177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:59.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.721+0000 7f07357af640 5 monclient: authenticate success, global_id 4604 2024-03-25T18:17:59.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.721+0000 7f07357af640 10 monclient: _renew_subs 2024-03-25T18:17:59.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.721+0000 7f07357af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:17:59.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.721+0000 7f07357af640 1 -- 172.21.15.64:0/3103818212 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07301e3690 con 0x7f0730145050 2024-03-25T18:17:59.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.721+0000 7f07357af640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:59.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.721+0000 7f07357af640 10 monclient: _renew_subs 2024-03-25T18:17:59.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.721+0000 7f07357af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:17:59.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.721+0000 7f07357af640 1 -- 172.21.15.64:0/3103818212 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07301e3b20 con 0x7f0730145050 2024-03-25T18:17:59.723 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.721+0000 7f07357af640 1 librados: init done 2024-03-25T18:17:59.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.722+0000 7f07357af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:59.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.722+0000 7f07357af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:59.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.722+0000 7f07357af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:17:59.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:17:59.722+0000 7f07357af640 1 -- 172.21.15.64:0/3103818212 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0730140bb0 con 0x7f0730145050 2024-03-25T18:18:00.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.040+0000 7f07177fe640 1 -- 172.21.15.64:0/3103818212 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0728003970 con 0x7f0730145050 2024-03-25T18:18:00.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.040+0000 7f07177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:00.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.040+0000 7f07177fe640 1 -- 172.21.15.64:0/3103818212 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0728003cc0 con 0x7f0730145050 2024-03-25T18:18:00.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.040+0000 7f07177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:00.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.040+0000 7f07177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:00.042 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.040+0000 7f07177fe640 10 monclient: dump: 2024-03-25T18:18:00.042 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:00.042 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:00.042 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:00.042 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:00.042 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:00.043 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:00.043 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:00.043 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:00.043 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:00.043 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:00.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.040+0000 7f072d7fa640 4 set_mon_vals no callback set 2024-03-25T18:18:00.043 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.041+0000 7f07177fe640 1 -- 172.21.15.64:0/3103818212 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f0728003ee0 con 0x7f0730145050 2024-03-25T18:18:00.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.042+0000 7f072effd640 10 monclient: get_auth_request con 0x7f071c06dd40 auth_method 0 2024-03-25T18:18:00.044 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.042+0000 7f07177fe640 10 client.4604.objecter ms_handle_connect 0x7f071c06dd40 2024-03-25T18:18:00.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.048+0000 7f07177fe640 1 -- 172.21.15.64:0/3103818212 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f0728054b00 con 0x7f0730145050 2024-03-25T18:18:00.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.048+0000 7f07177fe640 10 client.4604.objecter ms_dispatch 0x7f0730000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:00.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.048+0000 7f07177fe640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:00.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.048+0000 7f07177fe640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:00.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.048+0000 7f07177fe640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-25T18:18:00.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.048+0000 7f07177fe640 1 -- 172.21.15.64:0/3103818212 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0728050610 con 0x7f0730145050 2024-03-25T18:18:00.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.048+0000 7f07177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:00.049 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.048+0000 7f07177fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:00.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.261+0000 7f07357af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:00.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.261+0000 7f07357af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:00.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.261+0000 7f07357af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:18:00.262 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.261+0000 7f07357af640 1 -- 172.21.15.64:0/3103818212 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07300680d0 con 0x7f0730145050 2024-03-25T18:18:00.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.341+0000 7f07177fe640 1 -- 172.21.15.64:0/3103818212 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f0728052540 con 0x7f0730145050 2024-03-25T18:18:00.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.341+0000 7f07177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:00.343 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.341+0000 7f07177fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:00.343 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:18:00.343 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:00.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.344+0000 7f07357af640 10 librados: watch_flush enter 2024-03-25T18:18:00.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.344+0000 7f07357af640 10 librados: watch_flush exit 2024-03-25T18:18:00.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.344+0000 7f07357af640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-25T18:18:00.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.344+0000 7f07357af640 10 client.4604.objecter successfully canceled tick 2024-03-25T18:18:00.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.344+0000 7f07357af640 1 -- 172.21.15.64:0/3103818212 >> v1:172.21.15.64:6800/2915543638 conn(0x7f071c06dd40 legacy=0x7f071c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:00.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.344+0000 7f07357af640 10 monclient: shutdown 2024-03-25T18:18:00.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.344+0000 7f07357af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:00.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.344+0000 7f07357af640 1 -- 172.21.15.64:0/3103818212 >> v1:172.21.15.64:6790/0 conn(0x7f0730145050 legacy=0x7f07301362d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:00.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.344+0000 7f07357af640 1 -- 172.21.15.64:0/3103818212 shutdown_connections 2024-03-25T18:18:00.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.344+0000 7f07357af640 1 -- 172.21.15.64:0/3103818212 >> 172.21.15.64:0/3103818212 conn(0x7f073006c830 msgr2=0x7f07300707c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:00.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.345+0000 7f07357af640 1 -- 172.21.15.64:0/3103818212 shutdown_connections 2024-03-25T18:18:00.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.345+0000 7f07357af640 1 -- 172.21.15.64:0/3103818212 wait complete. 2024-03-25T18:18:00.346 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:00.345+0000 7f07357af640 1 librados: shutdown 2024-03-25T18:18:00.358 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:02.993 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:08:46.105 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:03.049 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:08:46.161 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:03.077 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:08:46.189 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:03.104 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:08:46.182 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:03.110 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:08:46.222 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:03.116 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:08:46.195 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:03.117 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:08:46.230 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:03.152 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:08:46.230 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:03.155 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:08:46.267 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:07.165 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:07.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.279+0000 7f94fb1b5640 10 monclient: get_monmap_and_config 2024-03-25T18:18:07.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.279+0000 7f94fb1b5640 10 monclient: build_initial_monmap 2024-03-25T18:18:07.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.279+0000 7f94fb1b5640 10 monclient: monmap: 2024-03-25T18:18:07.280 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:18:07.280 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:07.280 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:18:07.280380+0000 2024-03-25T18:18:07.280 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:18:07.280380+0000 2024-03-25T18:18:07.281 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:07.281 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:07.281 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:18:07.281 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:18:07.281 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:18:07.281 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:07.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.280+0000 7f94fb1b5640 1 Processor -- start 2024-03-25T18:18:07.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.280+0000 7f94fb1b5640 1 -- start start 2024-03-25T18:18:07.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.280+0000 7f94fb1b5640 10 monclient: init 2024-03-25T18:18:07.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.281+0000 7f94fb1b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:07.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.281+0000 7f94fb1b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:07.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.281+0000 7f94fb1b5640 10 monclient(hunting): picked mon.noname-a con 0x7f94f40b4d60 addr v1:172.21.15.64:6789/0 2024-03-25T18:18:07.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.281+0000 7f94fb1b5640 10 monclient(hunting): picked mon.noname-c con 0x7f94f40b1730 addr v1:172.21.15.86:6789/0 2024-03-25T18:18:07.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.281+0000 7f94fb1b5640 10 monclient(hunting): picked mon.noname-b con 0x7f94f414be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:18:07.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.281+0000 7f94fb1b5640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f40af940 con 0x7f94f40b4d60 2024-03-25T18:18:07.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.281+0000 7f94fb1b5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f40b0a70 con 0x7f94f40b1730 2024-03-25T18:18:07.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.281+0000 7f94fb1b5640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f40b0d50 con 0x7f94f414be90 2024-03-25T18:18:07.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.281+0000 7f94fb1b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:07.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.281+0000 7f94fb1b5640 10 monclient(hunting): authenticate will time out at 7473.520044s 2024-03-25T18:18:07.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.289+0000 7f94f9752640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f94f414be90 0x7f94f414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:54542/0 (socket says 172.21.15.64:54542) 2024-03-25T18:18:07.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.289+0000 7f94f9752640 1 -- 172.21.15.64:0/1449344320 learned_addr learned my addr 172.21.15.64:0/1449344320 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:18:07.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.301+0000 7f94f37fe640 1 -- 172.21.15.64:0/1449344320 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273693883 0 0) 0x7f94f40b0d50 con 0x7f94f414be90 2024-03-25T18:18:07.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.301+0000 7f94f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:07.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.301+0000 7f94f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:07.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.301+0000 7f94f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:07.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.301+0000 7f94f37fe640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:18:07.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.301+0000 7f94f37fe640 1 -- 172.21.15.64:0/1449344320 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94e4003520 con 0x7f94f414be90 2024-03-25T18:18:07.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.303+0000 7f94f37fe640 1 -- 172.21.15.64:0/1449344320 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833891093 0 0) 0x7f94f40af940 con 0x7f94f40b4d60 2024-03-25T18:18:07.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.303+0000 7f94f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:07.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.303+0000 7f94f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:07.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.303+0000 7f94f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:07.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.303+0000 7f94f37fe640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:18:07.305 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.303+0000 7f94f37fe640 1 -- 172.21.15.64:0/1449344320 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94f40b0d50 con 0x7f94f40b4d60 2024-03-25T18:18:07.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.311+0000 7f94f37fe640 1 -- 172.21.15.64:0/1449344320 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3689658324 0 0) 0x7f94e4003520 con 0x7f94f414be90 2024-03-25T18:18:07.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.311+0000 7f94f37fe640 1 -- 172.21.15.64:0/1449344320 >> v1:172.21.15.86:6789/0 conn(0x7f94f40b1730 legacy=0x7f94f40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.311+0000 7f94f37fe640 1 -- 172.21.15.64:0/1449344320 >> v1:172.21.15.64:6789/0 conn(0x7f94f40b4d60 legacy=0x7f94f413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.311+0000 7f94f37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:07.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.311+0000 7f94f37fe640 1 monclient: found mon.noname-b 2024-03-25T18:18:07.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.311+0000 7f94f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:07.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.311+0000 7f94f37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:18:07.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.311+0000 7f94f37fe640 1 -- 172.21.15.64:0/1449344320 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f40b1030 con 0x7f94f414be90 2024-03-25T18:18:07.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.311+0000 7f94f37fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:07.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.311+0000 7f94f37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:07.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.311+0000 7f94f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:07.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.311+0000 7f94f37fe640 1 -- 172.21.15.64:0/1449344320 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f94ec001a10 con 0x7f94f414be90 2024-03-25T18:18:07.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.311+0000 7f94f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:07.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.311+0000 7f94f37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:07.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.311+0000 7f94f37fe640 10 monclient: dump: 2024-03-25T18:18:07.313 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:07.313 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:07.313 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:07.313 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:07.313 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:07.313 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:07.314 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:07.314 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:07.314 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:07.314 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:07.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.311+0000 7f94fb1b5640 5 monclient: authenticate success, global_id 4607 2024-03-25T18:18:07.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.311+0000 7f94fb1b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:07.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.320+0000 7f94f37fe640 1 -- 172.21.15.64:0/1449344320 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94ec001b70 con 0x7f94f414be90 2024-03-25T18:18:07.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.320+0000 7f94f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:07.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.320+0000 7f94f37fe640 1 -- 172.21.15.64:0/1449344320 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f94ec001f80 con 0x7f94f414be90 2024-03-25T18:18:07.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.320+0000 7f94f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:07.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.320+0000 7f94f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:07.321 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.320+0000 7f94f37fe640 10 monclient: dump: 2024-03-25T18:18:07.321 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:07.321 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:07.321 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:07.321 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:07.321 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:07.321 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:07.321 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:07.321 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:07.322 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:07.322 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:07.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.320+0000 7f94fb1b5640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:07.322 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.320+0000 7f94fb1b5640 4 set_mon_vals no callback set 2024-03-25T18:18:07.323 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.320+0000 7f94fb1b5640 10 monclient: shutdown 2024-03-25T18:18:07.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.320+0000 7f94fb1b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:07.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.320+0000 7f94fb1b5640 1 -- 172.21.15.64:0/1449344320 >> v1:172.21.15.64:6790/0 conn(0x7f94f414be90 legacy=0x7f94f414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.321+0000 7f94fb1b5640 1 -- 172.21.15.64:0/1449344320 shutdown_connections 2024-03-25T18:18:07.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.321+0000 7f94fb1b5640 1 -- 172.21.15.64:0/1449344320 >> 172.21.15.64:0/1449344320 conn(0x7f94f40617a0 msgr2=0x7f94f4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:07.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.321+0000 7f94fb1b5640 1 -- 172.21.15.64:0/1449344320 shutdown_connections 2024-03-25T18:18:07.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.321+0000 7f94fb1b5640 1 -- 172.21.15.64:0/1449344320 wait complete. 2024-03-25T18:18:07.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.321+0000 7f94fb1b5640 10 monclient: build_initial_monmap 2024-03-25T18:18:07.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.321+0000 7f94fb1b5640 10 monclient: monmap: 2024-03-25T18:18:07.324 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:18:07.324 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:07.324 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:18:07.324 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:18:07.324 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:07.324 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:18:07.325 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:18:07.325 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:18:07.325 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:18:07.325 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:07.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.322+0000 7f94fb1b5640 1 librados: starting msgr at 2024-03-25T18:18:07.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.322+0000 7f94fb1b5640 1 librados: starting objecter 2024-03-25T18:18:07.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.322+0000 7f94fb1b5640 1 Processor -- start 2024-03-25T18:18:07.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.322+0000 7f94fb1b5640 1 -- start start 2024-03-25T18:18:07.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.322+0000 7f94fb1b5640 1 librados: setting wanted keys 2024-03-25T18:18:07.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.322+0000 7f94fb1b5640 1 librados: calling monclient init 2024-03-25T18:18:07.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.322+0000 7f94fb1b5640 10 monclient: init 2024-03-25T18:18:07.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.323+0000 7f94fb1b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:07.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.323+0000 7f94fb1b5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:07.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.323+0000 7f94fb1b5640 10 monclient(hunting): picked mon.noname-c con 0x7f94f40b1730 addr v1:172.21.15.64:6790/0 2024-03-25T18:18:07.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.323+0000 7f94fb1b5640 10 monclient(hunting): picked mon.noname-b con 0x7f94f40b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:18:07.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.323+0000 7f94fb1b5640 10 monclient(hunting): picked mon.noname-a con 0x7f94f414be90 addr v1:172.21.15.64:6789/0 2024-03-25T18:18:07.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.323+0000 7f94fb1b5640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f4141410 con 0x7f94f414be90 2024-03-25T18:18:07.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.323+0000 7f94fb1b5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f41383b0 con 0x7f94f40b4d60 2024-03-25T18:18:07.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.323+0000 7f94fb1b5640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f41388d0 con 0x7f94f40b1730 2024-03-25T18:18:07.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.323+0000 7f94fb1b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:07.327 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.323+0000 7f94fb1b5640 10 monclient(hunting): authenticate will time out at 7473.562250s 2024-03-25T18:18:07.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.335+0000 7f94f9752640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f94f414be90 0x7f94f41e7540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:35974/0 (socket says 172.21.15.64:35974) 2024-03-25T18:18:07.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.335+0000 7f94f9752640 1 -- 172.21.15.64:0/3475327282 learned_addr learned my addr 172.21.15.64:0/3475327282 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:18:07.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.337+0000 7f94f17fa640 10 client.?.objecter ms_handle_connect 0x7f94f414be90 2024-03-25T18:18:07.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.337+0000 7f94f17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:07.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.346+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110630923 0 0) 0x7f94f4141410 con 0x7f94f414be90 2024-03-25T18:18:07.347 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.346+0000 7f94f17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:07.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.346+0000 7f94f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:07.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.346+0000 7f94f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:07.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.346+0000 7f94f17fa640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:18:07.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.346+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94d8004eb0 con 0x7f94f414be90 2024-03-25T18:18:07.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.348+0000 7f94f17fa640 10 client.?.objecter ms_handle_connect 0x7f94f40b1730 2024-03-25T18:18:07.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.348+0000 7f94f17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:07.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.349+0000 7f94f17fa640 10 client.?.objecter ms_handle_connect 0x7f94f40b4d60 2024-03-25T18:18:07.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.349+0000 7f94f17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:07.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.353+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247752432 0 0) 0x7f94f41388d0 con 0x7f94f40b1730 2024-03-25T18:18:07.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.353+0000 7f94f17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:07.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.353+0000 7f94f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:07.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.353+0000 7f94f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:07.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.353+0000 7f94f17fa640 10 monclient(hunting): my global_id is 4610 2024-03-25T18:18:07.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.353+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94f4141410 con 0x7f94f40b1730 2024-03-25T18:18:07.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.354+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58541770 0 0) 0x7f94f41383b0 con 0x7f94f40b4d60 2024-03-25T18:18:07.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.354+0000 7f94f17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:07.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.354+0000 7f94f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:07.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.354+0000 7f94f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:07.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.354+0000 7f94f17fa640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:18:07.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.354+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94f41388d0 con 0x7f94f40b4d60 2024-03-25T18:18:07.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.364+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3241441898 0 0) 0x7f94f4141410 con 0x7f94f40b1730 2024-03-25T18:18:07.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.364+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94f41383b0 con 0x7f94f40b1730 2024-03-25T18:18:07.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.364+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f94e80018b0 con 0x7f94f40b1730 2024-03-25T18:18:07.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.364+0000 7f94f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:07.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.364+0000 7f94f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:07.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.364+0000 7f94f17fa640 10 monclient(hunting): dump: 2024-03-25T18:18:07.365 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:07.365 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:07.365 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:07.366 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:07.366 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:07.366 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:07.366 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:07.366 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:07.366 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:07.366 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:07.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.364+0000 7f94f17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:07.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.372+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1746105264 0 0) 0x7f94f41383b0 con 0x7f94f40b1730 2024-03-25T18:18:07.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 >> v1:172.21.15.86:6789/0 conn(0x7f94f40b4d60 legacy=0x7f94f41df1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 >> v1:172.21.15.64:6789/0 conn(0x7f94f414be90 legacy=0x7f94f41e7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94f17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:18:07.374 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94f17fa640 1 monclient: found mon.c 2024-03-25T18:18:07.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:07.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94f17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:18:07.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f4137e90 con 0x7f94f40b1730 2024-03-25T18:18:07.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94f17fa640 10 monclient: _finish_auth 0 2024-03-25T18:18:07.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94f17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:18:07.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:07.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94fb1b5640 5 monclient: authenticate success, global_id 4610 2024-03-25T18:18:07.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94fb1b5640 10 monclient: _renew_subs 2024-03-25T18:18:07.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94fb1b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:18:07.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94fb1b5640 1 -- 172.21.15.64:0/3475327282 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f4138140 con 0x7f94f40b1730 2024-03-25T18:18:07.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94fb1b5640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:07.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94fb1b5640 10 monclient: _renew_subs 2024-03-25T18:18:07.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94fb1b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:18:07.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94fb1b5640 1 -- 172.21.15.64:0/3475327282 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f4137bb0 con 0x7f94f40b1730 2024-03-25T18:18:07.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.373+0000 7f94fb1b5640 1 librados: init done 2024-03-25T18:18:07.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.374+0000 7f94fb1b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:07.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.374+0000 7f94fb1b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:07.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.374+0000 7f94fb1b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:18:07.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.374+0000 7f94fb1b5640 1 -- 172.21.15.64:0/3475327282 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f4141280 con 0x7f94f40b1730 2024-03-25T18:18:07.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.672+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94e8003610 con 0x7f94f40b1730 2024-03-25T18:18:07.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.672+0000 7f94f17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:07.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.672+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f94e8003920 con 0x7f94f40b1730 2024-03-25T18:18:07.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.672+0000 7f94f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:07.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.672+0000 7f94f17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:07.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.672+0000 7f94f17fa640 10 monclient: dump: 2024-03-25T18:18:07.673 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:07.674 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:07.674 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:07.674 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:07.674 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:07.674 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:07.674 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:07.674 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:07.674 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:07.674 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:07.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.672+0000 7f94f2ffd640 4 set_mon_vals no callback set 2024-03-25T18:18:07.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.672+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f94e80198f0 con 0x7f94f40b1730 2024-03-25T18:18:07.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.673+0000 7f94f3fff640 10 monclient: get_auth_request con 0x7f94d806ddf0 auth_method 0 2024-03-25T18:18:07.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.673+0000 7f94f17fa640 10 client.4610.objecter ms_handle_connect 0x7f94d806ddf0 2024-03-25T18:18:07.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.680+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f94e8053630 con 0x7f94f40b1730 2024-03-25T18:18:07.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.680+0000 7f94f17fa640 10 client.4610.objecter ms_dispatch 0x7f94f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:07.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.680+0000 7f94f17fa640 3 client.4610.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:07.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.680+0000 7f94f17fa640 3 client.4610.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:07.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.680+0000 7f94f17fa640 20 client.4610.objecter dump_active .. 0 homeless 2024-03-25T18:18:07.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.680+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f94e804ef70 con 0x7f94f40b1730 2024-03-25T18:18:07.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.680+0000 7f94f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:07.681 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.680+0000 7f94f17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:07.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.911+0000 7f94fb1b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:07.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.911+0000 7f94fb1b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:07.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.911+0000 7f94fb1b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:18:07.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.911+0000 7f94fb1b5640 1 -- 172.21.15.64:0/3475327282 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94f40627b0 con 0x7f94f40b1730 2024-03-25T18:18:07.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.937+0000 7f94f17fa640 1 -- 172.21.15.64:0/3475327282 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f94e8051070 con 0x7f94f40b1730 2024-03-25T18:18:07.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.937+0000 7f94f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:07.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.937+0000 7f94f17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:07.938 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:18:07.939 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:07.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.940+0000 7f94fb1b5640 10 librados: watch_flush enter 2024-03-25T18:18:07.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.940+0000 7f94fb1b5640 10 librados: watch_flush exit 2024-03-25T18:18:07.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.940+0000 7f94fb1b5640 20 client.4610.objecter shutdown clearing up homeless session... 2024-03-25T18:18:07.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.940+0000 7f94fb1b5640 10 client.4610.objecter successfully canceled tick 2024-03-25T18:18:07.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.940+0000 7f94fb1b5640 1 -- 172.21.15.64:0/3475327282 >> v1:172.21.15.64:6800/2915543638 conn(0x7f94d806ddf0 legacy=0x7f94d80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.940+0000 7f94fb1b5640 10 monclient: shutdown 2024-03-25T18:18:07.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.940+0000 7f94fb1b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:07.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.940+0000 7f94fb1b5640 1 -- 172.21.15.64:0/3475327282 >> v1:172.21.15.64:6790/0 conn(0x7f94f40b1730 legacy=0x7f94f4142be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.940+0000 7f94fb1b5640 1 -- 172.21.15.64:0/3475327282 shutdown_connections 2024-03-25T18:18:07.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.940+0000 7f94fb1b5640 1 -- 172.21.15.64:0/3475327282 >> 172.21.15.64:0/3475327282 conn(0x7f94f40617a0 msgr2=0x7f94f40b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:07.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.940+0000 7f94fb1b5640 1 -- 172.21.15.64:0/3475327282 shutdown_connections 2024-03-25T18:18:07.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.940+0000 7f94fb1b5640 1 -- 172.21.15.64:0/3475327282 wait complete. 2024-03-25T18:18:07.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:07.940+0000 7f94fb1b5640 1 librados: shutdown 2024-03-25T18:18:07.952 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:14.759 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:14.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.858+0000 7f9c6416c640 10 monclient: get_monmap_and_config 2024-03-25T18:18:14.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.858+0000 7f9c6416c640 10 monclient: build_initial_monmap 2024-03-25T18:18:14.860 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.858+0000 7f9c6416c640 10 monclient: monmap: 2024-03-25T18:18:14.860 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:18:14.860 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:14.860 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:18:14.859695+0000 2024-03-25T18:18:14.860 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:18:14.859695+0000 2024-03-25T18:18:14.860 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:14.860 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:14.860 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:18:14.860 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:18:14.860 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:18:14.860 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:14.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.859+0000 7f9c6416c640 1 Processor -- start 2024-03-25T18:18:14.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.859+0000 7f9c6416c640 1 -- start start 2024-03-25T18:18:14.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.859+0000 7f9c6416c640 10 monclient: init 2024-03-25T18:18:14.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.860+0000 7f9c6416c640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:14.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.860+0000 7f9c6416c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:14.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.860+0000 7f9c6416c640 10 monclient(hunting): picked mon.noname-c con 0x7f9c5c0b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:18:14.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.860+0000 7f9c6416c640 10 monclient(hunting): picked mon.noname-b con 0x7f9c5c0b1730 addr v1:172.21.15.64:6790/0 2024-03-25T18:18:14.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.860+0000 7f9c6416c640 10 monclient(hunting): picked mon.noname-a con 0x7f9c5c14be90 addr v1:172.21.15.64:6789/0 2024-03-25T18:18:14.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.860+0000 7f9c6416c640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c0af940 con 0x7f9c5c14be90 2024-03-25T18:18:14.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.860+0000 7f9c6416c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c0b0a70 con 0x7f9c5c0b4d60 2024-03-25T18:18:14.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.860+0000 7f9c6416c640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c0b0d50 con 0x7f9c5c0b1730 2024-03-25T18:18:14.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.860+0000 7f9c6416c640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:14.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.860+0000 7f9c6416c640 10 monclient(hunting): authenticate will time out at 7481.098739s 2024-03-25T18:18:14.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.870+0000 7f9c61f08640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f9c5c0b4d60 0x7f9c5c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:38466/0 (socket says 172.21.15.64:38466) 2024-03-25T18:18:14.871 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.870+0000 7f9c61f08640 1 -- 172.21.15.64:0/3633492716 learned_addr learned my addr 172.21.15.64:0/3633492716 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:18:14.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.887+0000 7f9c60f06640 1 -- 172.21.15.64:0/3633492716 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561279200 0 0) 0x7f9c5c0b0a70 con 0x7f9c5c0b4d60 2024-03-25T18:18:14.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.888+0000 7f9c60f06640 10 monclient(hunting): _negotiate 2024-03-25T18:18:14.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.888+0000 7f9c60f06640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:14.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.888+0000 7f9c60f06640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:14.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.888+0000 7f9c60f06640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:18:14.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.888+0000 7f9c60f06640 1 -- 172.21.15.64:0/3633492716 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c48003520 con 0x7f9c5c0b4d60 2024-03-25T18:18:14.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.902+0000 7f9c60f06640 1 -- 172.21.15.64:0/3633492716 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163953570 0 0) 0x7f9c5c0af940 con 0x7f9c5c14be90 2024-03-25T18:18:14.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.902+0000 7f9c60f06640 10 monclient(hunting): _negotiate 2024-03-25T18:18:14.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.902+0000 7f9c60f06640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:14.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.902+0000 7f9c60f06640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:14.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.902+0000 7f9c60f06640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:18:14.903 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.902+0000 7f9c60f06640 1 -- 172.21.15.64:0/3633492716 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c5c0b0a70 con 0x7f9c5c14be90 2024-03-25T18:18:14.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.902+0000 7f9c60f06640 1 -- 172.21.15.64:0/3633492716 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1863257543 0 0) 0x7f9c5c0b0d50 con 0x7f9c5c0b1730 2024-03-25T18:18:14.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.902+0000 7f9c60f06640 10 monclient(hunting): _negotiate 2024-03-25T18:18:14.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.902+0000 7f9c60f06640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:14.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.902+0000 7f9c60f06640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:14.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.902+0000 7f9c60f06640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:18:14.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.902+0000 7f9c60f06640 1 -- 172.21.15.64:0/3633492716 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c5c0af940 con 0x7f9c5c0b1730 2024-03-25T18:18:14.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.903+0000 7f9c60f06640 1 -- 172.21.15.64:0/3633492716 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 233339828 0 0) 0x7f9c48003520 con 0x7f9c5c0b4d60 2024-03-25T18:18:14.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.903+0000 7f9c60f06640 1 -- 172.21.15.64:0/3633492716 >> v1:172.21.15.64:6790/0 conn(0x7f9c5c0b1730 legacy=0x7f9c5c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:14.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.903+0000 7f9c60f06640 1 -- 172.21.15.64:0/3633492716 >> v1:172.21.15.64:6789/0 conn(0x7f9c5c14be90 legacy=0x7f9c5c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:14.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.903+0000 7f9c60f06640 10 monclient: _finish_hunting 0 2024-03-25T18:18:14.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.903+0000 7f9c60f06640 1 monclient: found mon.noname-c 2024-03-25T18:18:14.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.903+0000 7f9c60f06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:14.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.903+0000 7f9c60f06640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:18:14.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.903+0000 7f9c60f06640 1 -- 172.21.15.64:0/3633492716 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c0b1030 con 0x7f9c5c0b4d60 2024-03-25T18:18:14.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.903+0000 7f9c60f06640 10 monclient: _finish_auth 0 2024-03-25T18:18:14.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.903+0000 7f9c60f06640 10 monclient: _check_auth_tickets 2024-03-25T18:18:14.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.903+0000 7f9c60f06640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:14.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.903+0000 7f9c60f06640 1 -- 172.21.15.64:0/3633492716 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9c58001690 con 0x7f9c5c0b4d60 2024-03-25T18:18:14.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.903+0000 7f9c6416c640 5 monclient: authenticate success, global_id 4615 2024-03-25T18:18:14.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.903+0000 7f9c6416c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:14.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.903+0000 7f9c60f06640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:14.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.903+0000 7f9c60f06640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:14.906 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.903+0000 7f9c60f06640 10 monclient: dump: 2024-03-25T18:18:14.906 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:14.906 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:14.906 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:14.907 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:14.907 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:14.907 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:14.907 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:14.907 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:14.907 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:14.907 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:14.907 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.903+0000 7f9c6416c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:14.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.919+0000 7f9c60f06640 1 -- 172.21.15.64:0/3633492716 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c580017f0 con 0x7f9c5c0b4d60 2024-03-25T18:18:14.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.919+0000 7f9c60f06640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:14.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.919+0000 7f9c60f06640 1 -- 172.21.15.64:0/3633492716 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9c58001c00 con 0x7f9c5c0b4d60 2024-03-25T18:18:14.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.919+0000 7f9c6416c640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:14.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.919+0000 7f9c6416c640 4 set_mon_vals no callback set 2024-03-25T18:18:14.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.919+0000 7f9c60f06640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:14.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.919+0000 7f9c60f06640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:14.921 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.919+0000 7f9c60f06640 10 monclient: dump: 2024-03-25T18:18:14.921 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:14.921 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:14.921 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:14.921 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:14.921 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:14.921 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:14.921 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:14.921 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:14.921 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:14.921 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:14.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.920+0000 7f9c6416c640 10 monclient: shutdown 2024-03-25T18:18:14.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.920+0000 7f9c6416c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:14.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.920+0000 7f9c6416c640 1 -- 172.21.15.64:0/3633492716 >> v1:172.21.15.86:6789/0 conn(0x7f9c5c0b4d60 legacy=0x7f9c5c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:14.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.920+0000 7f9c6416c640 1 -- 172.21.15.64:0/3633492716 shutdown_connections 2024-03-25T18:18:14.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.920+0000 7f9c6416c640 1 -- 172.21.15.64:0/3633492716 >> 172.21.15.64:0/3633492716 conn(0x7f9c5c0617a0 msgr2=0x7f9c5c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:14.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.920+0000 7f9c6416c640 1 -- 172.21.15.64:0/3633492716 shutdown_connections 2024-03-25T18:18:14.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.920+0000 7f9c6416c640 1 -- 172.21.15.64:0/3633492716 wait complete. 2024-03-25T18:18:14.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.921+0000 7f9c6416c640 10 monclient: build_initial_monmap 2024-03-25T18:18:14.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.921+0000 7f9c6416c640 10 monclient: monmap: 2024-03-25T18:18:14.924 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:18:14.924 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:14.924 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:18:14.925 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:18:14.925 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:14.925 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:18:14.925 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:18:14.925 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:18:14.925 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:18:14.925 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:14.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.921+0000 7f9c6416c640 1 librados: starting msgr at 2024-03-25T18:18:14.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.921+0000 7f9c6416c640 1 librados: starting objecter 2024-03-25T18:18:14.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.921+0000 7f9c6416c640 1 Processor -- start 2024-03-25T18:18:14.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.921+0000 7f9c6416c640 1 -- start start 2024-03-25T18:18:14.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.921+0000 7f9c6416c640 1 librados: setting wanted keys 2024-03-25T18:18:14.925 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.921+0000 7f9c6416c640 1 librados: calling monclient init 2024-03-25T18:18:14.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.921+0000 7f9c6416c640 10 monclient: init 2024-03-25T18:18:14.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.922+0000 7f9c6416c640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:14.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.922+0000 7f9c6416c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:14.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.922+0000 7f9c6416c640 10 monclient(hunting): picked mon.noname-a con 0x7f9c5c0b1730 addr v1:172.21.15.64:6789/0 2024-03-25T18:18:14.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.922+0000 7f9c6416c640 10 monclient(hunting): picked mon.noname-b con 0x7f9c5c0b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:18:14.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.922+0000 7f9c6416c640 10 monclient(hunting): picked mon.noname-c con 0x7f9c5c14be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:18:14.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.922+0000 7f9c6416c640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c1d7730 con 0x7f9c5c0b1730 2024-03-25T18:18:14.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.922+0000 7f9c6416c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c13bdf0 con 0x7f9c5c0b4d60 2024-03-25T18:18:14.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.922+0000 7f9c6416c640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c142100 con 0x7f9c5c14be90 2024-03-25T18:18:14.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.922+0000 7f9c6416c640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:14.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.922+0000 7f9c6416c640 10 monclient(hunting): authenticate will time out at 7481.161038s 2024-03-25T18:18:14.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.931+0000 7f9c61f08640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f9c5c0b1730 0x7f9c5c1405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:40754/0 (socket says 172.21.15.64:40754) 2024-03-25T18:18:14.932 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.931+0000 7f9c61f08640 1 -- 172.21.15.64:0/3887875575 learned_addr learned my addr 172.21.15.64:0/3887875575 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:18:14.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.933+0000 7f9c527fc640 10 client.?.objecter ms_handle_connect 0x7f9c5c0b1730 2024-03-25T18:18:14.934 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.933+0000 7f9c527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:14.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.934+0000 7f9c527fc640 10 client.?.objecter ms_handle_connect 0x7f9c5c0b4d60 2024-03-25T18:18:14.935 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.934+0000 7f9c527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:14.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.935+0000 7f9c527fc640 10 client.?.objecter ms_handle_connect 0x7f9c5c14be90 2024-03-25T18:18:14.936 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.935+0000 7f9c527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:14.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.940+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977364675 0 0) 0x7f9c5c142100 con 0x7f9c5c14be90 2024-03-25T18:18:14.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.940+0000 7f9c527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:14.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.940+0000 7f9c527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:14.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.940+0000 7f9c527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:14.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.940+0000 7f9c527fc640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:18:14.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.940+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c40004eb0 con 0x7f9c5c14be90 2024-03-25T18:18:14.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.940+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857832769 0 0) 0x7f9c5c1d7730 con 0x7f9c5c0b1730 2024-03-25T18:18:14.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.940+0000 7f9c527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:14.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.940+0000 7f9c527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:14.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.940+0000 7f9c527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:14.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.940+0000 7f9c527fc640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:18:14.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.940+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c5c142100 con 0x7f9c5c0b1730 2024-03-25T18:18:14.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.950+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255261295 0 0) 0x7f9c5c13bdf0 con 0x7f9c5c0b4d60 2024-03-25T18:18:14.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.950+0000 7f9c527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:14.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.950+0000 7f9c527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:14.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.950+0000 7f9c527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:14.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.950+0000 7f9c527fc640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:18:14.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.950+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c5c1d7730 con 0x7f9c5c0b4d60 2024-03-25T18:18:14.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.957+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1284737055 0 0) 0x7f9c5c142100 con 0x7f9c5c0b1730 2024-03-25T18:18:14.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.957+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c5c13bdf0 con 0x7f9c5c0b1730 2024-03-25T18:18:14.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.957+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9c58001b40 con 0x7f9c5c0b1730 2024-03-25T18:18:14.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.957+0000 7f9c527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:14.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.957+0000 7f9c527fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:14.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.957+0000 7f9c527fc640 10 monclient(hunting): dump: 2024-03-25T18:18:14.959 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:14.959 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:14.959 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:14.959 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:14.959 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:14.959 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:14.959 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:14.959 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:14.959 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:14.959 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:14.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.957+0000 7f9c527fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:14.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.959+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 890231546 0 0) 0x7f9c40004eb0 con 0x7f9c5c14be90 2024-03-25T18:18:14.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.959+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c5c142100 con 0x7f9c5c14be90 2024-03-25T18:18:14.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.959+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9c54001bd0 con 0x7f9c5c14be90 2024-03-25T18:18:14.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.959+0000 7f9c527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:14.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.959+0000 7f9c527fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:14.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.959+0000 7f9c527fc640 10 monclient(hunting): dump: 2024-03-25T18:18:14.961 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:14.961 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:14.961 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:14.961 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:14.961 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:14.961 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:14.961 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:14.961 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:14.961 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:14.961 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:14.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.961+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2800707 0 0) 0x7f9c5c1d7730 con 0x7f9c5c0b4d60 2024-03-25T18:18:14.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.961+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c40004eb0 con 0x7f9c5c0b4d60 2024-03-25T18:18:14.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.961+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9c4c001830 con 0x7f9c5c0b4d60 2024-03-25T18:18:14.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.961+0000 7f9c527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:14.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.961+0000 7f9c527fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:14.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.961+0000 7f9c527fc640 10 monclient(hunting): dump: 2024-03-25T18:18:14.962 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:14.962 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:14.962 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:14.963 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:14.963 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:14.963 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:14.963 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:14.963 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:14.963 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:14.963 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:14.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2346139615 0 0) 0x7f9c5c13bdf0 con 0x7f9c5c0b1730 2024-03-25T18:18:14.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 >> v1:172.21.15.64:6790/0 conn(0x7f9c5c14be90 legacy=0x7f9c5c1d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:14.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 >> v1:172.21.15.86:6789/0 conn(0x7f9c5c0b4d60 legacy=0x7f9c5c13b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:14.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c527fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:14.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c527fc640 1 monclient: found mon.a 2024-03-25T18:18:14.966 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:14.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c527fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:18:14.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c142290 con 0x7f9c5c0b1730 2024-03-25T18:18:14.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c527fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:14.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c527fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:14.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:14.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c6416c640 5 monclient: authenticate success, global_id 4614 2024-03-25T18:18:14.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c6416c640 10 monclient: _renew_subs 2024-03-25T18:18:14.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c6416c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:18:14.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c6416c640 1 -- 172.21.15.64:0/3887875575 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5c1d8320 con 0x7f9c5c0b1730 2024-03-25T18:18:14.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c6416c640 10 client.4614.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:14.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c6416c640 10 monclient: _renew_subs 2024-03-25T18:18:14.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c6416c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:18:14.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c6416c640 1 -- 172.21.15.64:0/3887875575 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5c1d85d0 con 0x7f9c5c0b1730 2024-03-25T18:18:14.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.965+0000 7f9c6416c640 1 librados: init done 2024-03-25T18:18:14.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.966+0000 7f9c6416c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:14.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.966+0000 7f9c6416c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:14.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.966+0000 7f9c6416c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:18:14.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:14.966+0000 7f9c6416c640 1 -- 172.21.15.64:0/3887875575 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c5c0627b0 con 0x7f9c5c0b1730 2024-03-25T18:18:15.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.271+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c580011f0 con 0x7f9c5c0b1730 2024-03-25T18:18:15.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.271+0000 7f9c527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:15.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.271+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9c58004c30 con 0x7f9c5c0b1730 2024-03-25T18:18:15.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.271+0000 7f9c527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:15.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.271+0000 7f9c527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:15.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.271+0000 7f9c527fc640 10 monclient: dump: 2024-03-25T18:18:15.273 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:15.273 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:15.273 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:15.273 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:15.273 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:15.273 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:15.273 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:15.273 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:15.273 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:15.273 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:15.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.271+0000 7f9c53fff640 4 set_mon_vals no callback set 2024-03-25T18:18:15.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.272+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f9c5801ac80 con 0x7f9c5c0b1730 2024-03-25T18:18:15.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.272+0000 7f9c61707640 10 monclient: get_auth_request con 0x7f9c4006e0b0 auth_method 0 2024-03-25T18:18:15.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.272+0000 7f9c527fc640 10 client.4614.objecter ms_handle_connect 0x7f9c4006e0b0 2024-03-25T18:18:15.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.278+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f9c580548e0 con 0x7f9c5c0b1730 2024-03-25T18:18:15.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.278+0000 7f9c527fc640 10 client.4614.objecter ms_dispatch 0x7f9c5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:15.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.278+0000 7f9c527fc640 3 client.4614.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:15.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.278+0000 7f9c527fc640 3 client.4614.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:15.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.278+0000 7f9c527fc640 20 client.4614.objecter dump_active .. 0 homeless 2024-03-25T18:18:15.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.278+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c58050220 con 0x7f9c5c0b1730 2024-03-25T18:18:15.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.278+0000 7f9c527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:15.279 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.278+0000 7f9c527fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:15.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.551+0000 7f9c6416c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:15.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.551+0000 7f9c6416c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:15.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.551+0000 7f9c6416c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:18:15.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.551+0000 7f9c6416c640 1 -- 172.21.15.64:0/3887875575 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c5c140170 con 0x7f9c5c0b1730 2024-03-25T18:18:15.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.578+0000 7f9c527fc640 1 -- 172.21.15.64:0/3887875575 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f9c58052320 con 0x7f9c5c0b1730 2024-03-25T18:18:15.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.578+0000 7f9c527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:15.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.578+0000 7f9c527fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:15.579 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:18:15.579 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:15.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.580+0000 7f9c6416c640 10 librados: watch_flush enter 2024-03-25T18:18:15.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.580+0000 7f9c6416c640 10 librados: watch_flush exit 2024-03-25T18:18:15.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.580+0000 7f9c6416c640 20 client.4614.objecter shutdown clearing up homeless session... 2024-03-25T18:18:15.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.580+0000 7f9c6416c640 10 client.4614.objecter successfully canceled tick 2024-03-25T18:18:15.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.580+0000 7f9c6416c640 1 -- 172.21.15.64:0/3887875575 >> v1:172.21.15.64:6800/2915543638 conn(0x7f9c4006e0b0 legacy=0x7f9c40070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:15.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.580+0000 7f9c6416c640 10 monclient: shutdown 2024-03-25T18:18:15.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.580+0000 7f9c6416c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:15.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.580+0000 7f9c6416c640 1 -- 172.21.15.64:0/3887875575 >> v1:172.21.15.64:6789/0 conn(0x7f9c5c0b1730 legacy=0x7f9c5c1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:15.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.580+0000 7f9c6416c640 1 -- 172.21.15.64:0/3887875575 shutdown_connections 2024-03-25T18:18:15.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.580+0000 7f9c6416c640 1 -- 172.21.15.64:0/3887875575 >> 172.21.15.64:0/3887875575 conn(0x7f9c5c0617a0 msgr2=0x7f9c5c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:15.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.580+0000 7f9c6416c640 1 -- 172.21.15.64:0/3887875575 shutdown_connections 2024-03-25T18:18:15.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.580+0000 7f9c6416c640 1 -- 172.21.15.64:0/3887875575 wait complete. 2024-03-25T18:18:15.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:15.580+0000 7f9c6416c640 1 librados: shutdown 2024-03-25T18:18:15.594 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:22.400 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:22.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.513+0000 7f942f11c640 10 monclient: get_monmap_and_config 2024-03-25T18:18:22.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.513+0000 7f942f11c640 10 monclient: build_initial_monmap 2024-03-25T18:18:22.514 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.513+0000 7f942f11c640 10 monclient: monmap: 2024-03-25T18:18:22.514 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:18:22.515 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:22.515 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:18:22.514664+0000 2024-03-25T18:18:22.515 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:18:22.514664+0000 2024-03-25T18:18:22.515 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:22.515 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:22.515 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:18:22.515 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:18:22.515 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:18:22.515 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:22.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.514+0000 7f942f11c640 1 Processor -- start 2024-03-25T18:18:22.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.514+0000 7f942f11c640 1 -- start start 2024-03-25T18:18:22.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.514+0000 7f942f11c640 10 monclient: init 2024-03-25T18:18:22.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.515+0000 7f942f11c640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:22.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.515+0000 7f942f11c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:22.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.515+0000 7f942f11c640 10 monclient(hunting): picked mon.noname-a con 0x7f94280a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:18:22.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.515+0000 7f942f11c640 10 monclient(hunting): picked mon.noname-b con 0x7f942813f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:18:22.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.515+0000 7f942f11c640 10 monclient(hunting): picked mon.noname-c con 0x7f94281430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:18:22.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.515+0000 7f942f11c640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9428142520 con 0x7f94280a98f0 2024-03-25T18:18:22.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.515+0000 7f942f11c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94280b28a0 con 0x7f94281430c0 2024-03-25T18:18:22.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.515+0000 7f942f11c640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94280b2a30 con 0x7f942813f710 2024-03-25T18:18:22.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.515+0000 7f942f11c640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:22.517 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.515+0000 7f942f11c640 10 monclient(hunting): authenticate will time out at 7488.754142s 2024-03-25T18:18:22.525 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.524+0000 7f942d6b9640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f94281430c0 0x7f942814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:47726/0 (socket says 172.21.15.64:47726) 2024-03-25T18:18:22.525 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.524+0000 7f942d6b9640 1 -- 172.21.15.64:0/4214969866 learned_addr learned my addr 172.21.15.64:0/4214969866 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:18:22.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.544+0000 7f94277fe640 1 -- 172.21.15.64:0/4214969866 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453824760 0 0) 0x7f9428142520 con 0x7f94280a98f0 2024-03-25T18:18:22.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.544+0000 7f94277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.544+0000 7f94277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.544+0000 7f94277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.544+0000 7f94277fe640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:18:22.545 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.544+0000 7f94277fe640 1 -- 172.21.15.64:0/4214969866 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9418003520 con 0x7f94280a98f0 2024-03-25T18:18:22.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.547+0000 7f94277fe640 1 -- 172.21.15.64:0/4214969866 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 772876567 0 0) 0x7f94280b2a30 con 0x7f942813f710 2024-03-25T18:18:22.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.547+0000 7f94277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.547+0000 7f94277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.547+0000 7f94277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.548 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.547+0000 7f94277fe640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:18:22.549 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.547+0000 7f94277fe640 1 -- 172.21.15.64:0/4214969866 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9428142520 con 0x7f942813f710 2024-03-25T18:18:22.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.549+0000 7f94277fe640 1 -- 172.21.15.64:0/4214969866 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372098032 0 0) 0x7f94280b28a0 con 0x7f94281430c0 2024-03-25T18:18:22.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.549+0000 7f94277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.549+0000 7f94277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.549+0000 7f94277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.549+0000 7f94277fe640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:18:22.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.549+0000 7f94277fe640 1 -- 172.21.15.64:0/4214969866 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94280b2a30 con 0x7f94281430c0 2024-03-25T18:18:22.551 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.550+0000 7f94277fe640 1 -- 172.21.15.64:0/4214969866 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3184112596 0 0) 0x7f9418003520 con 0x7f94280a98f0 2024-03-25T18:18:22.551 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.550+0000 7f94277fe640 1 -- 172.21.15.64:0/4214969866 >> v1:172.21.15.64:6790/0 conn(0x7f942813f710 legacy=0x7f94281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.552 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.550+0000 7f94277fe640 1 -- 172.21.15.64:0/4214969866 >> v1:172.21.15.86:6789/0 conn(0x7f94281430c0 legacy=0x7f942814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.552 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.550+0000 7f94277fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:22.552 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.550+0000 7f94277fe640 1 monclient: found mon.noname-a 2024-03-25T18:18:22.552 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.550+0000 7f94277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:22.552 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.550+0000 7f94277fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:18:22.552 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.550+0000 7f94277fe640 1 -- 172.21.15.64:0/4214969866 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94280b44c0 con 0x7f94280a98f0 2024-03-25T18:18:22.552 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.550+0000 7f94277fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:22.552 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.550+0000 7f94277fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:22.552 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.550+0000 7f94277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:22.552 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.550+0000 7f942f11c640 5 monclient: authenticate success, global_id 4617 2024-03-25T18:18:22.552 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.550+0000 7f942f11c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:22.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.551+0000 7f94277fe640 1 -- 172.21.15.64:0/4214969866 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f941c001650 con 0x7f94280a98f0 2024-03-25T18:18:22.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.551+0000 7f94277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:22.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.551+0000 7f94277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:22.553 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.551+0000 7f94277fe640 10 monclient: dump: 2024-03-25T18:18:22.553 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:22.553 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:22.553 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:22.553 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:22.553 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:22.554 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:22.554 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:22.554 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:22.554 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:22.554 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:22.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.552+0000 7f942f11c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:22.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.557+0000 7f94277fe640 1 -- 172.21.15.64:0/4214969866 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f941c0018a0 con 0x7f94280a98f0 2024-03-25T18:18:22.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.557+0000 7f94277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:22.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.557+0000 7f94277fe640 1 -- 172.21.15.64:0/4214969866 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f941c001c30 con 0x7f94280a98f0 2024-03-25T18:18:22.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.557+0000 7f942f11c640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:22.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.557+0000 7f942f11c640 4 set_mon_vals no callback set 2024-03-25T18:18:22.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.557+0000 7f94277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:22.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.557+0000 7f94277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:22.558 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.557+0000 7f94277fe640 10 monclient: dump: 2024-03-25T18:18:22.558 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:22.559 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:22.559 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:22.559 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:22.559 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:22.559 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:22.559 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:22.559 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:22.559 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:22.559 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:22.561 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.557+0000 7f942f11c640 10 monclient: shutdown 2024-03-25T18:18:22.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.558+0000 7f942f11c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:22.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.558+0000 7f942f11c640 1 -- 172.21.15.64:0/4214969866 >> v1:172.21.15.64:6789/0 conn(0x7f94280a98f0 legacy=0x7f94280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.558+0000 7f942f11c640 1 -- 172.21.15.64:0/4214969866 shutdown_connections 2024-03-25T18:18:22.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.558+0000 7f942f11c640 1 -- 172.21.15.64:0/4214969866 >> 172.21.15.64:0/4214969866 conn(0x7f94280b7ea0 msgr2=0x7f94280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:22.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.558+0000 7f942f11c640 1 -- 172.21.15.64:0/4214969866 shutdown_connections 2024-03-25T18:18:22.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.558+0000 7f942f11c640 1 -- 172.21.15.64:0/4214969866 wait complete. 2024-03-25T18:18:22.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.558+0000 7f942f11c640 10 monclient: build_initial_monmap 2024-03-25T18:18:22.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.558+0000 7f942f11c640 10 monclient: monmap: 2024-03-25T18:18:22.562 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:18:22.562 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:22.562 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:18:22.562 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:18:22.562 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:22.562 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:18:22.562 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:18:22.563 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:18:22.563 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:18:22.563 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:22.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.559+0000 7f942f11c640 1 librados: starting msgr at 2024-03-25T18:18:22.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.559+0000 7f942f11c640 1 librados: starting objecter 2024-03-25T18:18:22.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.559+0000 7f942f11c640 1 Processor -- start 2024-03-25T18:18:22.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.559+0000 7f942f11c640 1 -- start start 2024-03-25T18:18:22.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.559+0000 7f942f11c640 1 librados: setting wanted keys 2024-03-25T18:18:22.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.559+0000 7f942f11c640 1 librados: calling monclient init 2024-03-25T18:18:22.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.559+0000 7f942f11c640 10 monclient: init 2024-03-25T18:18:22.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.560+0000 7f942f11c640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:22.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.560+0000 7f942f11c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:22.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.560+0000 7f942f11c640 10 monclient(hunting): picked mon.noname-a con 0x7f94280a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:18:22.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.560+0000 7f942f11c640 10 monclient(hunting): picked mon.noname-c con 0x7f942813f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:18:22.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.560+0000 7f942f11c640 10 monclient(hunting): picked mon.noname-b con 0x7f94281430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:18:22.563 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.560+0000 7f942f11c640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94280b17e0 con 0x7f94280a98f0 2024-03-25T18:18:22.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.560+0000 7f942f11c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94281392b0 con 0x7f94281430c0 2024-03-25T18:18:22.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.560+0000 7f942f11c640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94280abca0 con 0x7f942813f710 2024-03-25T18:18:22.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.560+0000 7f942f11c640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:22.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.560+0000 7f942f11c640 10 monclient(hunting): authenticate will time out at 7488.798578s 2024-03-25T18:18:22.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.578+0000 7f942d6b9640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f94281430c0 0x7f94280ac610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:47734/0 (socket says 172.21.15.64:47734) 2024-03-25T18:18:22.579 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.578+0000 7f942d6b9640 1 -- 172.21.15.64:0/3476485564 learned_addr learned my addr 172.21.15.64:0/3476485564 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:18:22.580 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.580+0000 7f94257fa640 10 client.?.objecter ms_handle_connect 0x7f94281430c0 2024-03-25T18:18:22.581 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.580+0000 7f94257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:22.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.585+0000 7f94257fa640 10 client.?.objecter ms_handle_connect 0x7f94280a98f0 2024-03-25T18:18:22.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.585+0000 7f94257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:22.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.586+0000 7f94257fa640 1 -- 172.21.15.64:0/3476485564 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645316287 0 0) 0x7f94281392b0 con 0x7f94281430c0 2024-03-25T18:18:22.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.586+0000 7f94257fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.586+0000 7f94257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.586+0000 7f94257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.586+0000 7f94257fa640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:18:22.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.586+0000 7f94257fa640 1 -- 172.21.15.64:0/3476485564 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f940c004eb0 con 0x7f94281430c0 2024-03-25T18:18:22.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.599+0000 7f94257fa640 1 -- 172.21.15.64:0/3476485564 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 396797759 0 0) 0x7f940c004eb0 con 0x7f94281430c0 2024-03-25T18:18:22.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.599+0000 7f94257fa640 1 -- 172.21.15.64:0/3476485564 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94281392b0 con 0x7f94281430c0 2024-03-25T18:18:22.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.599+0000 7f94257fa640 1 -- 172.21.15.64:0/3476485564 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9420001bd0 con 0x7f94281430c0 2024-03-25T18:18:22.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.599+0000 7f94257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:22.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.599+0000 7f94257fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:22.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.599+0000 7f94257fa640 10 monclient(hunting): dump: 2024-03-25T18:18:22.601 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:22.601 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:22.601 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:22.601 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:22.601 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:22.602 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:22.602 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:22.602 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:22.602 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:22.602 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:22.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.599+0000 7f94257fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:22.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.600+0000 7f94257fa640 1 -- 172.21.15.64:0/3476485564 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863589930 0 0) 0x7f94280b44c0 con 0x7f94280a98f0 2024-03-25T18:18:22.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.600+0000 7f94257fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.600+0000 7f94257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.600+0000 7f94257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.600+0000 7f94257fa640 10 monclient(hunting): my global_id is 4620 2024-03-25T18:18:22.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.600+0000 7f94257fa640 1 -- 172.21.15.64:0/3476485564 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f940c004eb0 con 0x7f94280a98f0 2024-03-25T18:18:22.606 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f94257fa640 1 -- 172.21.15.64:0/3476485564 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4115941229 0 0) 0x7f94281392b0 con 0x7f94281430c0 2024-03-25T18:18:22.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f94257fa640 1 -- 172.21.15.64:0/3476485564 >> v1:172.21.15.64:6790/0 conn(0x7f942813f710 legacy=0x7f94281366b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f94257fa640 1 -- 172.21.15.64:0/3476485564 >> v1:172.21.15.64:6789/0 conn(0x7f94280a98f0 legacy=0x7f9428138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f94257fa640 10 monclient: _finish_hunting 0 2024-03-25T18:18:22.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f94257fa640 1 monclient: found mon.b 2024-03-25T18:18:22.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f94257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:22.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f94257fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:18:22.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f94257fa640 1 -- 172.21.15.64:0/3476485564 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94280abf10 con 0x7f94281430c0 2024-03-25T18:18:22.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f94257fa640 10 monclient: _finish_auth 0 2024-03-25T18:18:22.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f94257fa640 10 monclient: _check_auth_tickets 2024-03-25T18:18:22.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f94257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:22.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f942f11c640 5 monclient: authenticate success, global_id 4624 2024-03-25T18:18:22.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f942f11c640 10 monclient: _renew_subs 2024-03-25T18:18:22.607 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f942f11c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:18:22.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f942f11c640 1 -- 172.21.15.64:0/3476485564 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94280ac1f0 con 0x7f94281430c0 2024-03-25T18:18:22.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f942f11c640 10 client.4624.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:22.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f942f11c640 10 monclient: _renew_subs 2024-03-25T18:18:22.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f942f11c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:18:22.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f942f11c640 1 -- 172.21.15.64:0/3476485564 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94281e34b0 con 0x7f94281430c0 2024-03-25T18:18:22.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.605+0000 7f942f11c640 1 librados: init done 2024-03-25T18:18:22.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.606+0000 7f942f11c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:22.608 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.606+0000 7f942f11c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:22.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.606+0000 7f942f11c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:18:22.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.606+0000 7f942f11c640 1 -- 172.21.15.64:0/3476485564 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94280627b0 con 0x7f94281430c0 2024-03-25T18:18:22.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.894+0000 7f94257fa640 1 -- 172.21.15.64:0/3476485564 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9420004b60 con 0x7f94281430c0 2024-03-25T18:18:22.895 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.894+0000 7f94257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:22.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.894+0000 7f94257fa640 1 -- 172.21.15.64:0/3476485564 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9420002100 con 0x7f94281430c0 2024-03-25T18:18:22.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.894+0000 7f94257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:22.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.894+0000 7f94257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:22.896 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.894+0000 7f94257fa640 10 monclient: dump: 2024-03-25T18:18:22.896 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:22.896 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:22.896 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:22.896 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:22.896 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:22.896 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:22.896 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:22.896 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:22.896 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:22.896 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:22.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.894+0000 7f9426ffd640 4 set_mon_vals no callback set 2024-03-25T18:18:22.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.895+0000 7f94257fa640 1 -- 172.21.15.64:0/3476485564 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f9420005030 con 0x7f94281430c0 2024-03-25T18:18:22.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.896+0000 7f942ceb8640 10 monclient: get_auth_request con 0x7f940c06dd20 auth_method 0 2024-03-25T18:18:22.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.896+0000 7f94257fa640 10 client.4624.objecter ms_handle_connect 0x7f940c06dd20 2024-03-25T18:18:22.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.900+0000 7f94257fa640 1 -- 172.21.15.64:0/3476485564 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f9420054b90 con 0x7f94281430c0 2024-03-25T18:18:22.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.901+0000 7f94257fa640 10 client.4624.objecter ms_dispatch 0x7f9428000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:22.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.901+0000 7f94257fa640 3 client.4624.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:22.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.901+0000 7f94257fa640 3 client.4624.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:22.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.901+0000 7f94257fa640 20 client.4624.objecter dump_active .. 0 homeless 2024-03-25T18:18:22.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.901+0000 7f94257fa640 1 -- 172.21.15.64:0/3476485564 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f94200506a0 con 0x7f94281430c0 2024-03-25T18:18:22.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.901+0000 7f94257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:22.902 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:22.901+0000 7f94257fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:23.051 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:06.164 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:23.102 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:06.215 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:23.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.118+0000 7f942f11c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:23.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.118+0000 7f942f11c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:23.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.118+0000 7f942f11c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:18:23.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.118+0000 7f942f11c640 1 -- 172.21.15.64:0/3476485564 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94280b2300 con 0x7f94281430c0 2024-03-25T18:18:23.133 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:06.245 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:23.154 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:06.266 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:23.174 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:06.287 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:23.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.190+0000 7f94257fa640 1 -- 172.21.15.64:0/3476485564 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f94200525d0 con 0x7f94281430c0 2024-03-25T18:18:23.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.190+0000 7f94257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:23.191 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.190+0000 7f94257fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:23.191 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:18:23.192 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:23.192 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:09:06.270 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:23.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.192+0000 7f942f11c640 10 librados: watch_flush enter 2024-03-25T18:18:23.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.192+0000 7f942f11c640 10 librados: watch_flush exit 2024-03-25T18:18:23.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.192+0000 7f942f11c640 20 client.4624.objecter shutdown clearing up homeless session... 2024-03-25T18:18:23.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.192+0000 7f942f11c640 10 client.4624.objecter successfully canceled tick 2024-03-25T18:18:23.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.192+0000 7f942f11c640 1 -- 172.21.15.64:0/3476485564 >> v1:172.21.15.64:6800/2915543638 conn(0x7f940c06dd20 legacy=0x7f940c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:23.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.192+0000 7f942f11c640 10 monclient: shutdown 2024-03-25T18:18:23.194 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.192+0000 7f942f11c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:23.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.192+0000 7f942f11c640 1 -- 172.21.15.64:0/3476485564 >> v1:172.21.15.86:6789/0 conn(0x7f94281430c0 legacy=0x7f94280ac610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:23.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.193+0000 7f942f11c640 1 -- 172.21.15.64:0/3476485564 shutdown_connections 2024-03-25T18:18:23.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.193+0000 7f942f11c640 1 -- 172.21.15.64:0/3476485564 >> 172.21.15.64:0/3476485564 conn(0x7f94280b7ea0 msgr2=0x7f9428074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:23.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.193+0000 7f942f11c640 1 -- 172.21.15.64:0/3476485564 shutdown_connections 2024-03-25T18:18:23.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.193+0000 7f942f11c640 1 -- 172.21.15.64:0/3476485564 wait complete. 2024-03-25T18:18:23.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:23.193+0000 7f942f11c640 1 librados: shutdown 2024-03-25T18:18:23.195 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:09:06.273 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:23.205 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:23.206 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:06.319 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:23.225 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:09:06.303 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:30.010 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:30.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.120+0000 7f8a9e04a640 10 monclient: get_monmap_and_config 2024-03-25T18:18:30.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.120+0000 7f8a9e04a640 10 monclient: build_initial_monmap 2024-03-25T18:18:30.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.120+0000 7f8a9e04a640 10 monclient: monmap: 2024-03-25T18:18:30.123 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:18:30.123 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:30.123 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:18:30.122469+0000 2024-03-25T18:18:30.123 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:18:30.122469+0000 2024-03-25T18:18:30.123 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:30.123 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:30.123 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:18:30.123 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:18:30.123 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:18:30.123 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:30.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.121+0000 7f8a9e04a640 1 Processor -- start 2024-03-25T18:18:30.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.121+0000 7f8a9e04a640 1 -- start start 2024-03-25T18:18:30.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.121+0000 7f8a9e04a640 10 monclient: init 2024-03-25T18:18:30.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.122+0000 7f8a9e04a640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:30.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.122+0000 7f8a9e04a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:30.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.122+0000 7f8a9e04a640 10 monclient(hunting): picked mon.noname-b con 0x7f8a980a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:18:30.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.122+0000 7f8a9e04a640 10 monclient(hunting): picked mon.noname-a con 0x7f8a9813f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:18:30.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.122+0000 7f8a9e04a640 10 monclient(hunting): picked mon.noname-c con 0x7f8a981430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:18:30.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.122+0000 7f8a9e04a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a98142520 con 0x7f8a9813f710 2024-03-25T18:18:30.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.122+0000 7f8a9e04a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a980b28a0 con 0x7f8a981430c0 2024-03-25T18:18:30.125 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.122+0000 7f8a9e04a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a980b2a30 con 0x7f8a980a98f0 2024-03-25T18:18:30.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.122+0000 7f8a9e04a640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:30.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.122+0000 7f8a9e04a640 10 monclient(hunting): authenticate will time out at 7496.362398s 2024-03-25T18:18:30.134 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.132+0000 7f8a97fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f8a981430c0 0x7f8a9814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:55626/0 (socket says 172.21.15.64:55626) 2024-03-25T18:18:30.134 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.132+0000 7f8a97fff640 1 -- 172.21.15.64:0/4111364285 learned_addr learned my addr 172.21.15.64:0/4111364285 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:18:30.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.143+0000 7f8a967fc640 1 -- 172.21.15.64:0/4111364285 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457202683 0 0) 0x7f8a980b28a0 con 0x7f8a981430c0 2024-03-25T18:18:30.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.143+0000 7f8a967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:30.145 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.143+0000 7f8a967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:30.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.143+0000 7f8a967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:30.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.143+0000 7f8a967fc640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:18:30.146 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.143+0000 7f8a967fc640 1 -- 172.21.15.64:0/4111364285 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a80003520 con 0x7f8a981430c0 2024-03-25T18:18:30.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.158+0000 7f8a967fc640 1 -- 172.21.15.64:0/4111364285 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232690419 0 0) 0x7f8a98142520 con 0x7f8a9813f710 2024-03-25T18:18:30.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.158+0000 7f8a967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:30.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.158+0000 7f8a967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:30.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.158+0000 7f8a967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:30.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.158+0000 7f8a967fc640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:18:30.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.158+0000 7f8a967fc640 1 -- 172.21.15.64:0/4111364285 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a980b28a0 con 0x7f8a9813f710 2024-03-25T18:18:30.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.166+0000 7f8a967fc640 1 -- 172.21.15.64:0/4111364285 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1706654742 0 0) 0x7f8a80003520 con 0x7f8a981430c0 2024-03-25T18:18:30.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.166+0000 7f8a967fc640 1 -- 172.21.15.64:0/4111364285 >> v1:172.21.15.64:6790/0 conn(0x7f8a980a98f0 legacy=0x7f8a980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.166+0000 7f8a967fc640 1 -- 172.21.15.64:0/4111364285 >> v1:172.21.15.64:6789/0 conn(0x7f8a9813f710 legacy=0x7f8a981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.166+0000 7f8a967fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:30.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.166+0000 7f8a967fc640 1 monclient: found mon.noname-c 2024-03-25T18:18:30.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.166+0000 7f8a967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:30.168 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.166+0000 7f8a967fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:18:30.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.166+0000 7f8a967fc640 1 -- 172.21.15.64:0/4111364285 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a980b44c0 con 0x7f8a981430c0 2024-03-25T18:18:30.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.166+0000 7f8a967fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:30.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.166+0000 7f8a967fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:30.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.166+0000 7f8a967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:30.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.166+0000 7f8a9e04a640 5 monclient: authenticate success, global_id 4627 2024-03-25T18:18:30.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.166+0000 7f8a967fc640 1 -- 172.21.15.64:0/4111364285 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f8a90001a10 con 0x7f8a981430c0 2024-03-25T18:18:30.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.166+0000 7f8a967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:30.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.166+0000 7f8a967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:30.169 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.166+0000 7f8a967fc640 10 monclient: dump: 2024-03-25T18:18:30.169 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:30.169 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:30.169 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:30.169 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:30.169 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:30.169 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:30.169 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:30.170 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:30.170 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:30.170 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:30.170 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.166+0000 7f8a9e04a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:30.175 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.173+0000 7f8a967fc640 1 -- 172.21.15.64:0/4111364285 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a90001b70 con 0x7f8a981430c0 2024-03-25T18:18:30.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.173+0000 7f8a967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:30.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.173+0000 7f8a967fc640 1 -- 172.21.15.64:0/4111364285 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f8a90001f80 con 0x7f8a981430c0 2024-03-25T18:18:30.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.173+0000 7f8a967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:30.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.173+0000 7f8a967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:30.176 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.173+0000 7f8a967fc640 10 monclient: dump: 2024-03-25T18:18:30.176 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:30.176 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:30.176 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:30.176 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:30.176 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:30.176 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:30.176 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:30.176 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:30.176 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:30.176 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:30.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.173+0000 7f8a9e04a640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:30.177 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.174+0000 7f8a9e04a640 4 set_mon_vals no callback set 2024-03-25T18:18:30.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.174+0000 7f8a9e04a640 10 monclient: shutdown 2024-03-25T18:18:30.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.174+0000 7f8a9e04a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:30.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.174+0000 7f8a9e04a640 1 -- 172.21.15.64:0/4111364285 >> v1:172.21.15.86:6789/0 conn(0x7f8a981430c0 legacy=0x7f8a9814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.174+0000 7f8a9e04a640 1 -- 172.21.15.64:0/4111364285 shutdown_connections 2024-03-25T18:18:30.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.174+0000 7f8a9e04a640 1 -- 172.21.15.64:0/4111364285 >> 172.21.15.64:0/4111364285 conn(0x7f8a980b7ea0 msgr2=0x7f8a980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:30.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.174+0000 7f8a9e04a640 1 -- 172.21.15.64:0/4111364285 shutdown_connections 2024-03-25T18:18:30.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.174+0000 7f8a9e04a640 1 -- 172.21.15.64:0/4111364285 wait complete. 2024-03-25T18:18:30.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.175+0000 7f8a9e04a640 10 monclient: build_initial_monmap 2024-03-25T18:18:30.179 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.175+0000 7f8a9e04a640 10 monclient: monmap: 2024-03-25T18:18:30.180 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:18:30.180 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:30.180 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:18:30.180 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:18:30.180 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:30.180 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:18:30.180 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:18:30.180 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:18:30.180 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:18:30.180 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:30.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.176+0000 7f8a9e04a640 1 librados: starting msgr at 2024-03-25T18:18:30.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.176+0000 7f8a9e04a640 1 librados: starting objecter 2024-03-25T18:18:30.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.176+0000 7f8a9e04a640 1 Processor -- start 2024-03-25T18:18:30.180 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.176+0000 7f8a9e04a640 1 -- start start 2024-03-25T18:18:30.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.176+0000 7f8a9e04a640 1 librados: setting wanted keys 2024-03-25T18:18:30.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.176+0000 7f8a9e04a640 1 librados: calling monclient init 2024-03-25T18:18:30.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.176+0000 7f8a9e04a640 10 monclient: init 2024-03-25T18:18:30.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.177+0000 7f8a9e04a640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:30.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.177+0000 7f8a9e04a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:30.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.177+0000 7f8a9e04a640 10 monclient(hunting): picked mon.noname-a con 0x7f8a980a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:18:30.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.177+0000 7f8a9e04a640 10 monclient(hunting): picked mon.noname-c con 0x7f8a9813f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:18:30.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.177+0000 7f8a9e04a640 10 monclient(hunting): picked mon.noname-b con 0x7f8a981430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:18:30.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.177+0000 7f8a9e04a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a98135230 con 0x7f8a980a98f0 2024-03-25T18:18:30.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.177+0000 7f8a9e04a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a98135da0 con 0x7f8a981430c0 2024-03-25T18:18:30.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.177+0000 7f8a9e04a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a98135f30 con 0x7f8a9813f710 2024-03-25T18:18:30.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.177+0000 7f8a9e04a640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:30.181 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.177+0000 7f8a9e04a640 10 monclient(hunting): authenticate will time out at 7496.416856s 2024-03-25T18:18:30.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.187+0000 7f8a977fe640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f8a980a98f0 0x7f8a981d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:54270/0 (socket says 172.21.15.64:54270) 2024-03-25T18:18:30.189 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.187+0000 7f8a977fe640 1 -- 172.21.15.64:0/2445752513 learned_addr learned my addr 172.21.15.64:0/2445752513 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:18:30.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.188+0000 7f8a7ffff640 10 client.?.objecter ms_handle_connect 0x7f8a980a98f0 2024-03-25T18:18:30.190 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.188+0000 7f8a7ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:30.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.193+0000 7f8a7ffff640 10 client.?.objecter ms_handle_connect 0x7f8a9813f710 2024-03-25T18:18:30.195 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.193+0000 7f8a7ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:30.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.197+0000 7f8a7ffff640 10 client.?.objecter ms_handle_connect 0x7f8a981430c0 2024-03-25T18:18:30.199 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.197+0000 7f8a7ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:30.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.202+0000 7f8a97fff640 1 -- 172.21.15.64:0/2445752513 >> v1:172.21.15.86:6789/0 conn(0x7f8a981430c0 legacy=0x7f8a981db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:18:30.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.202+0000 7f8a97fff640 1 -- 172.21.15.64:0/2445752513 >> v1:172.21.15.86:6789/0 conn(0x7f8a981430c0 legacy=0x7f8a981db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:18:30.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.202+0000 7f8a97fff640 1 --1- 172.21.15.64:0/2445752513 >> v1:172.21.15.86:6789/0 conn(0x7f8a981430c0 0x7f8a981db560 :-1 s=OPENED pgs=65619 cs=1 l=1).handle_message read tag failed 2024-03-25T18:18:30.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.202+0000 7f8a97fff640 1 --1- 172.21.15.64:0/2445752513 >> v1:172.21.15.86:6789/0 conn(0x7f8a981430c0 0x7f8a981db560 :-1 s=OPENED pgs=65619 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:18:30.204 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.202+0000 7f8a7ffff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.86:6789/0 2024-03-25T18:18:30.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.207+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777879909 0 0) 0x7f8a98135f30 con 0x7f8a9813f710 2024-03-25T18:18:30.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.207+0000 7f8a7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:30.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.207+0000 7f8a7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:30.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.207+0000 7f8a7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:30.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.207+0000 7f8a7ffff640 10 monclient(hunting): my global_id is 4625 2024-03-25T18:18:30.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.207+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a84004eb0 con 0x7f8a9813f710 2024-03-25T18:18:30.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.207+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558573904 0 0) 0x7f8a98135230 con 0x7f8a980a98f0 2024-03-25T18:18:30.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.207+0000 7f8a7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:30.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.207+0000 7f8a7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:30.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.207+0000 7f8a7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:30.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.207+0000 7f8a7ffff640 10 monclient(hunting): my global_id is 4626 2024-03-25T18:18:30.210 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.207+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a98135f30 con 0x7f8a980a98f0 2024-03-25T18:18:30.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.219+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2573017179 0 0) 0x7f8a98135f30 con 0x7f8a980a98f0 2024-03-25T18:18:30.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.219+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a98135230 con 0x7f8a980a98f0 2024-03-25T18:18:30.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.219+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f8a88001970 con 0x7f8a980a98f0 2024-03-25T18:18:30.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.219+0000 7f8a7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:30.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.219+0000 7f8a7ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:30.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.219+0000 7f8a7ffff640 10 monclient(hunting): dump: 2024-03-25T18:18:30.221 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:30.221 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:30.222 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:30.222 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:30.222 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:30.222 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:30.222 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:30.222 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:30.222 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:30.222 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:30.222 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.219+0000 7f8a7ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:30.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.232+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1624699575 0 0) 0x7f8a84004eb0 con 0x7f8a9813f710 2024-03-25T18:18:30.234 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.232+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a98135f30 con 0x7f8a9813f710 2024-03-25T18:18:30.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.232+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f8a8c001930 con 0x7f8a9813f710 2024-03-25T18:18:30.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.232+0000 7f8a7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:30.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.232+0000 7f8a7ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:30.235 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.232+0000 7f8a7ffff640 10 monclient(hunting): dump: 2024-03-25T18:18:30.235 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:30.235 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:30.235 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:30.235 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:30.235 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:30.235 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:30.235 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:30.235 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:30.235 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:30.235 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.236+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2121561006 0 0) 0x7f8a98135230 con 0x7f8a980a98f0 2024-03-25T18:18:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.236+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 >> v1:172.21.15.64:6790/0 conn(0x7f8a9813f710 legacy=0x7f8a981dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.236+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 >> v1:172.21.15.86:6789/0 conn(0x7f8a981430c0 legacy=0x7f8a981db560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:18:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.236+0000 7f8a7ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.236+0000 7f8a7ffff640 1 monclient: found mon.a 2024-03-25T18:18:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.236+0000 7f8a7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:30.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.236+0000 7f8a7ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:18:30.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.236+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a981360c0 con 0x7f8a980a98f0 2024-03-25T18:18:30.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.236+0000 7f8a7ffff640 10 monclient: _finish_auth 0 2024-03-25T18:18:30.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.236+0000 7f8a7ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:30.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.236+0000 7f8a7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:30.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.236+0000 7f8a9e04a640 5 monclient: authenticate success, global_id 4626 2024-03-25T18:18:30.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.236+0000 7f8a9e04a640 10 monclient: _renew_subs 2024-03-25T18:18:30.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.236+0000 7f8a9e04a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:18:30.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.236+0000 7f8a9e04a640 1 -- 172.21.15.64:0/2445752513 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a981d5000 con 0x7f8a980a98f0 2024-03-25T18:18:30.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.237+0000 7f8a9e04a640 10 client.4626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:30.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.237+0000 7f8a9e04a640 10 monclient: _renew_subs 2024-03-25T18:18:30.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.237+0000 7f8a9e04a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:18:30.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.237+0000 7f8a9e04a640 1 -- 172.21.15.64:0/2445752513 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a981d5350 con 0x7f8a980a98f0 2024-03-25T18:18:30.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.237+0000 7f8a9e04a640 1 librados: init done 2024-03-25T18:18:30.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.238+0000 7f8a9e04a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:30.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.238+0000 7f8a9e04a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:30.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.238+0000 7f8a9e04a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:18:30.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.238+0000 7f8a9e04a640 1 -- 172.21.15.64:0/2445752513 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a98135090 con 0x7f8a980a98f0 2024-03-25T18:18:30.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.562+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a88003610 con 0x7f8a980a98f0 2024-03-25T18:18:30.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.562+0000 7f8a7ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:30.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.562+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f8a880037c0 con 0x7f8a980a98f0 2024-03-25T18:18:30.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.562+0000 7f8a7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:30.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.562+0000 7f8a7ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:30.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.562+0000 7f8a7ffff640 10 monclient: dump: 2024-03-25T18:18:30.565 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:30.565 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:30.565 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:30.565 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:30.565 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:30.565 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:30.566 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:30.566 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:30.566 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:30.566 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:30.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.562+0000 7f8a95ffb640 4 set_mon_vals no callback set 2024-03-25T18:18:30.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.563+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f8a880039e0 con 0x7f8a980a98f0 2024-03-25T18:18:30.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.564+0000 7f8a96ffd640 10 monclient: get_auth_request con 0x7f8a8406de80 auth_method 0 2024-03-25T18:18:30.567 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.564+0000 7f8a7ffff640 10 client.4626.objecter ms_handle_connect 0x7f8a8406de80 2024-03-25T18:18:30.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.571+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f8a88054630 con 0x7f8a980a98f0 2024-03-25T18:18:30.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.571+0000 7f8a7ffff640 10 client.4626.objecter ms_dispatch 0x7f8a98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:30.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.571+0000 7f8a7ffff640 3 client.4626.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:30.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.571+0000 7f8a7ffff640 3 client.4626.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:30.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.572+0000 7f8a7ffff640 20 client.4626.objecter dump_active .. 0 homeless 2024-03-25T18:18:30.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.572+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a88050140 con 0x7f8a980a98f0 2024-03-25T18:18:30.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.572+0000 7f8a7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:30.574 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.572+0000 7f8a7ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:30.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.824+0000 7f8a9e04a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:30.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.824+0000 7f8a9e04a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:30.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.824+0000 7f8a9e04a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:18:30.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.824+0000 7f8a9e04a640 1 -- 172.21.15.64:0/2445752513 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a9813b700 con 0x7f8a980a98f0 2024-03-25T18:18:30.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.870+0000 7f8a7ffff640 1 -- 172.21.15.64:0/2445752513 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f8a88052070 con 0x7f8a980a98f0 2024-03-25T18:18:30.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.870+0000 7f8a7ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:30.872 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.870+0000 7f8a7ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:30.872 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:18:30.872 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:30.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.872+0000 7f8a9e04a640 10 librados: watch_flush enter 2024-03-25T18:18:30.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.872+0000 7f8a9e04a640 10 librados: watch_flush exit 2024-03-25T18:18:30.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.872+0000 7f8a9e04a640 20 client.4626.objecter shutdown clearing up homeless session... 2024-03-25T18:18:30.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.872+0000 7f8a9e04a640 10 client.4626.objecter successfully canceled tick 2024-03-25T18:18:30.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.872+0000 7f8a9e04a640 1 -- 172.21.15.64:0/2445752513 >> v1:172.21.15.64:6800/2915543638 conn(0x7f8a8406de80 legacy=0x7f8a84070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.872+0000 7f8a9e04a640 10 monclient: shutdown 2024-03-25T18:18:30.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.872+0000 7f8a9e04a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:30.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.872+0000 7f8a9e04a640 1 -- 172.21.15.64:0/2445752513 >> v1:172.21.15.64:6789/0 conn(0x7f8a980a98f0 legacy=0x7f8a981d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.872+0000 7f8a9e04a640 1 -- 172.21.15.64:0/2445752513 shutdown_connections 2024-03-25T18:18:30.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.872+0000 7f8a9e04a640 1 -- 172.21.15.64:0/2445752513 >> 172.21.15.64:0/2445752513 conn(0x7f8a980b7ea0 msgr2=0x7f8a9814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:30.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.872+0000 7f8a9e04a640 1 -- 172.21.15.64:0/2445752513 shutdown_connections 2024-03-25T18:18:30.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.872+0000 7f8a9e04a640 1 -- 172.21.15.64:0/2445752513 wait complete. 2024-03-25T18:18:30.875 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:30.872+0000 7f8a9e04a640 1 librados: shutdown 2024-03-25T18:18:30.886 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:37.693 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:37.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.809+0000 7f78190a1640 10 monclient: get_monmap_and_config 2024-03-25T18:18:37.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.809+0000 7f78190a1640 10 monclient: build_initial_monmap 2024-03-25T18:18:37.810 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.809+0000 7f78190a1640 10 monclient: monmap: 2024-03-25T18:18:37.811 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:18:37.811 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:37.811 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:18:37.810666+0000 2024-03-25T18:18:37.811 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:18:37.810666+0000 2024-03-25T18:18:37.811 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:37.811 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:37.811 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:18:37.811 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:18:37.811 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:18:37.811 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:37.812 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.810+0000 7f78190a1640 1 Processor -- start 2024-03-25T18:18:37.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.810+0000 7f78190a1640 1 -- start start 2024-03-25T18:18:37.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.810+0000 7f78190a1640 10 monclient: init 2024-03-25T18:18:37.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.811+0000 7f78190a1640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:37.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.811+0000 7f78190a1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:37.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.811+0000 7f78190a1640 10 monclient(hunting): picked mon.noname-b con 0x7f78140b4d60 addr v1:172.21.15.64:6790/0 2024-03-25T18:18:37.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.811+0000 7f78190a1640 10 monclient(hunting): picked mon.noname-a con 0x7f78140b1730 addr v1:172.21.15.64:6789/0 2024-03-25T18:18:37.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.811+0000 7f78190a1640 10 monclient(hunting): picked mon.noname-c con 0x7f781414be90 addr v1:172.21.15.86:6789/0 2024-03-25T18:18:37.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.811+0000 7f78190a1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78140af940 con 0x7f78140b1730 2024-03-25T18:18:37.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.811+0000 7f78190a1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78140b0a70 con 0x7f781414be90 2024-03-25T18:18:37.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.811+0000 7f78190a1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78140b0d50 con 0x7f78140b4d60 2024-03-25T18:18:37.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.811+0000 7f78190a1640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:37.813 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.811+0000 7f78190a1640 10 monclient(hunting): authenticate will time out at 7504.049726s 2024-03-25T18:18:37.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.826+0000 7f781359e640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f781414be90 0x7f781414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:46442/0 (socket says 172.21.15.64:46442) 2024-03-25T18:18:37.827 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.826+0000 7f781359e640 1 -- 172.21.15.64:0/3859515619 learned_addr learned my addr 172.21.15.64:0/3859515619 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:18:37.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.839+0000 7f7811d9b640 1 -- 172.21.15.64:0/3859515619 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924595470 0 0) 0x7f78140b0d50 con 0x7f78140b4d60 2024-03-25T18:18:37.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.839+0000 7f7811d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:37.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.839+0000 7f7811d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:37.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.839+0000 7f7811d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:37.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.839+0000 7f7811d9b640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:18:37.841 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.839+0000 7f7811d9b640 1 -- 172.21.15.64:0/3859515619 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77fc003520 con 0x7f78140b4d60 2024-03-25T18:18:37.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.846+0000 7f7811d9b640 1 -- 172.21.15.64:0/3859515619 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025202676 0 0) 0x7f78140b0a70 con 0x7f781414be90 2024-03-25T18:18:37.847 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.846+0000 7f7811d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:37.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.846+0000 7f7811d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:37.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.846+0000 7f7811d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:37.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.846+0000 7f7811d9b640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:18:37.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.846+0000 7f7811d9b640 1 -- 172.21.15.64:0/3859515619 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78140b0d50 con 0x7f781414be90 2024-03-25T18:18:37.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.846+0000 7f7811d9b640 1 -- 172.21.15.64:0/3859515619 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483332210 0 0) 0x7f78140af940 con 0x7f78140b1730 2024-03-25T18:18:37.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.846+0000 7f7811d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:37.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.846+0000 7f7811d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:37.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.846+0000 7f7811d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:37.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.846+0000 7f7811d9b640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:18:37.848 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.846+0000 7f7811d9b640 1 -- 172.21.15.64:0/3859515619 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78140b0a70 con 0x7f78140b1730 2024-03-25T18:18:37.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.849+0000 7f7811d9b640 1 -- 172.21.15.64:0/3859515619 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2428897936 0 0) 0x7f77fc003520 con 0x7f78140b4d60 2024-03-25T18:18:37.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.849+0000 7f7811d9b640 1 -- 172.21.15.64:0/3859515619 >> v1:172.21.15.86:6789/0 conn(0x7f781414be90 legacy=0x7f781414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:37.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.849+0000 7f7811d9b640 1 -- 172.21.15.64:0/3859515619 >> v1:172.21.15.64:6789/0 conn(0x7f78140b1730 legacy=0x7f78140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:37.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.849+0000 7f7811d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:18:37.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.849+0000 7f7811d9b640 1 monclient: found mon.noname-b 2024-03-25T18:18:37.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.849+0000 7f7811d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:37.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.849+0000 7f7811d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:18:37.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.849+0000 7f7811d9b640 1 -- 172.21.15.64:0/3859515619 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78140b1030 con 0x7f78140b4d60 2024-03-25T18:18:37.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.849+0000 7f7811d9b640 10 monclient: _finish_auth 0 2024-03-25T18:18:37.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.849+0000 7f7811d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:18:37.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.849+0000 7f7811d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:37.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.849+0000 7f7811d9b640 1 -- 172.21.15.64:0/3859515619 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f7804001690 con 0x7f78140b4d60 2024-03-25T18:18:37.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.849+0000 7f78190a1640 5 monclient: authenticate success, global_id 4628 2024-03-25T18:18:37.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.849+0000 7f78190a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:37.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.849+0000 7f7811d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:37.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.849+0000 7f7811d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:37.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.849+0000 7f7811d9b640 10 monclient: dump: 2024-03-25T18:18:37.851 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:37.852 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:37.852 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:37.852 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:37.852 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:37.852 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:37.852 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:37.852 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:37.852 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:37.852 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:37.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.849+0000 7f78190a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:37.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.856+0000 7f7811d9b640 1 -- 172.21.15.64:0/3859515619 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78040017f0 con 0x7f78140b4d60 2024-03-25T18:18:37.857 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.856+0000 7f7811d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:37.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.856+0000 7f7811d9b640 1 -- 172.21.15.64:0/3859515619 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f7804001c00 con 0x7f78140b4d60 2024-03-25T18:18:37.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.856+0000 7f7811d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:37.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.856+0000 7f7811d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:37.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.856+0000 7f7811d9b640 10 monclient: dump: 2024-03-25T18:18:37.858 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:37.858 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:37.858 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:37.858 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:37.858 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:37.858 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:37.858 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:37.858 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:37.859 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:37.859 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:37.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.856+0000 7f78190a1640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:37.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.856+0000 7f78190a1640 4 set_mon_vals no callback set 2024-03-25T18:18:37.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.857+0000 7f78190a1640 10 monclient: shutdown 2024-03-25T18:18:37.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.857+0000 7f78190a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:37.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.857+0000 7f78190a1640 1 -- 172.21.15.64:0/3859515619 >> v1:172.21.15.64:6790/0 conn(0x7f78140b4d60 legacy=0x7f781413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:37.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.857+0000 7f78190a1640 1 -- 172.21.15.64:0/3859515619 shutdown_connections 2024-03-25T18:18:37.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.857+0000 7f78190a1640 1 -- 172.21.15.64:0/3859515619 >> 172.21.15.64:0/3859515619 conn(0x7f78140617a0 msgr2=0x7f7814061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:37.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.857+0000 7f78190a1640 1 -- 172.21.15.64:0/3859515619 shutdown_connections 2024-03-25T18:18:37.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.857+0000 7f78190a1640 1 -- 172.21.15.64:0/3859515619 wait complete. 2024-03-25T18:18:37.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.858+0000 7f78190a1640 10 monclient: build_initial_monmap 2024-03-25T18:18:37.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.858+0000 7f78190a1640 10 monclient: monmap: 2024-03-25T18:18:37.862 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:18:37.862 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:37.862 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:18:37.862 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:18:37.862 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:37.862 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:18:37.862 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:18:37.862 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:18:37.862 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:18:37.862 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:37.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.858+0000 7f78190a1640 1 librados: starting msgr at 2024-03-25T18:18:37.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.859+0000 7f78190a1640 1 librados: starting objecter 2024-03-25T18:18:37.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.859+0000 7f78190a1640 1 Processor -- start 2024-03-25T18:18:37.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.859+0000 7f78190a1640 1 -- start start 2024-03-25T18:18:37.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.859+0000 7f78190a1640 1 librados: setting wanted keys 2024-03-25T18:18:37.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.859+0000 7f78190a1640 1 librados: calling monclient init 2024-03-25T18:18:37.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.859+0000 7f78190a1640 10 monclient: init 2024-03-25T18:18:37.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.860+0000 7f78190a1640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:37.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.860+0000 7f78190a1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:37.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.860+0000 7f78190a1640 10 monclient(hunting): picked mon.noname-a con 0x7f78140b1730 addr v1:172.21.15.64:6789/0 2024-03-25T18:18:37.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.860+0000 7f78190a1640 10 monclient(hunting): picked mon.noname-c con 0x7f78140b4d60 addr v1:172.21.15.64:6790/0 2024-03-25T18:18:37.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.860+0000 7f78190a1640 10 monclient(hunting): picked mon.noname-b con 0x7f781414be90 addr v1:172.21.15.86:6789/0 2024-03-25T18:18:37.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.860+0000 7f78190a1640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78141d7980 con 0x7f78140b1730 2024-03-25T18:18:37.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.860+0000 7f78190a1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78141420a0 con 0x7f781414be90 2024-03-25T18:18:37.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.860+0000 7f78190a1640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78141432c0 con 0x7f78140b4d60 2024-03-25T18:18:37.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.860+0000 7f78190a1640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:37.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.860+0000 7f78190a1640 10 monclient(hunting): authenticate will time out at 7504.098773s 2024-03-25T18:18:37.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.867+0000 7f781259c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f78140b4d60 0x7f7814138360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:37840/0 (socket says 172.21.15.64:37840) 2024-03-25T18:18:37.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.867+0000 7f781259c640 1 -- 172.21.15.64:0/1169180595 learned_addr learned my addr 172.21.15.64:0/1169180595 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:18:37.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.868+0000 7f77fb7fe640 10 client.?.objecter ms_handle_connect 0x7f78140b4d60 2024-03-25T18:18:37.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.868+0000 7f77fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:37.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.872+0000 7f77fb7fe640 10 client.?.objecter ms_handle_connect 0x7f781414be90 2024-03-25T18:18:37.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.872+0000 7f77fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:37.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.880+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308473813 0 0) 0x7f78141420a0 con 0x7f781414be90 2024-03-25T18:18:37.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.880+0000 7f77fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:37.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.880+0000 7f77fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:37.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.880+0000 7f77fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:37.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.880+0000 7f77fb7fe640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:18:37.881 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.880+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7800004eb0 con 0x7f781414be90 2024-03-25T18:18:37.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.884+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355277265 0 0) 0x7f78141432c0 con 0x7f78140b4d60 2024-03-25T18:18:37.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.884+0000 7f77fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:37.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.884+0000 7f77fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:37.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.884+0000 7f77fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:37.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.884+0000 7f77fb7fe640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:18:37.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.884+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78141420a0 con 0x7f78140b4d60 2024-03-25T18:18:37.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.884+0000 7f77fb7fe640 10 client.?.objecter ms_handle_connect 0x7f78140b1730 2024-03-25T18:18:37.885 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.884+0000 7f77fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:37.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.896+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2209958071 0 0) 0x7f7800004eb0 con 0x7f781414be90 2024-03-25T18:18:37.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.896+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78141432c0 con 0x7f781414be90 2024-03-25T18:18:37.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.896+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f780c001bd0 con 0x7f781414be90 2024-03-25T18:18:37.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.896+0000 7f77fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:37.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.896+0000 7f77fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:37.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.896+0000 7f77fb7fe640 10 monclient(hunting): dump: 2024-03-25T18:18:37.897 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:37.898 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:37.898 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:37.898 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:37.898 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:37.898 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:37.898 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:37.898 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:37.898 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:37.898 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:37.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.896+0000 7f77fb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:37.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.897+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 42881575 0 0) 0x7f78141420a0 con 0x7f78140b4d60 2024-03-25T18:18:37.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.897+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7800004eb0 con 0x7f78140b4d60 2024-03-25T18:18:37.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.897+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f78080019f0 con 0x7f78140b4d60 2024-03-25T18:18:37.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.897+0000 7f77fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:37.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.897+0000 7f77fb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:37.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.897+0000 7f77fb7fe640 10 monclient(hunting): dump: 2024-03-25T18:18:37.899 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:37.899 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:37.899 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:37.900 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:37.900 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:37.900 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:37.900 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:37.900 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:37.900 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:37.900 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:37.900 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.899+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507733748 0 0) 0x7f78141d7980 con 0x7f78140b1730 2024-03-25T18:18:37.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.899+0000 7f77fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:37.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.899+0000 7f77fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:37.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.899+0000 7f77fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:37.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.899+0000 7f77fb7fe640 10 monclient(hunting): my global_id is 4632 2024-03-25T18:18:37.901 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.899+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78141420a0 con 0x7f78140b1730 2024-03-25T18:18:37.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.909+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2510978578 0 0) 0x7f78141420a0 con 0x7f78140b1730 2024-03-25T18:18:37.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.909+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78141d7980 con 0x7f78140b1730 2024-03-25T18:18:37.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.909+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f7804001b00 con 0x7f78140b1730 2024-03-25T18:18:37.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.909+0000 7f77fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:37.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.909+0000 7f77fb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:37.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.909+0000 7f77fb7fe640 10 monclient(hunting): dump: 2024-03-25T18:18:37.912 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:37.912 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:37.912 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:37.912 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:37.912 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:37.912 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:37.912 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:37.912 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:37.912 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:37.912 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:37.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3549281446 0 0) 0x7f7800004eb0 con 0x7f78140b4d60 2024-03-25T18:18:37.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 >> v1:172.21.15.86:6789/0 conn(0x7f781414be90 legacy=0x7f78141d7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:37.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 >> v1:172.21.15.64:6789/0 conn(0x7f78140b1730 legacy=0x7f7814141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:37.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f77fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:37.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f77fb7fe640 1 monclient: found mon.c 2024-03-25T18:18:37.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f77fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:37.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f77fb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:18:37.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7814143940 con 0x7f78140b4d60 2024-03-25T18:18:37.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f77fb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:37.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f77fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:37.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f77fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:37.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f78190a1640 5 monclient: authenticate success, global_id 4631 2024-03-25T18:18:37.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f78190a1640 10 monclient: _renew_subs 2024-03-25T18:18:37.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f78190a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:18:37.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f78190a1640 1 -- 172.21.15.64:0/1169180595 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7814138a70 con 0x7f78140b4d60 2024-03-25T18:18:37.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f78190a1640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:37.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f78190a1640 10 monclient: _renew_subs 2024-03-25T18:18:37.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f78190a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:18:37.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f78190a1640 1 -- 172.21.15.64:0/1169180595 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f78141d8700 con 0x7f78140b4d60 2024-03-25T18:18:37.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.910+0000 7f78190a1640 1 librados: init done 2024-03-25T18:18:37.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.911+0000 7f78190a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:37.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.911+0000 7f78190a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:37.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.911+0000 7f78190a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:18:37.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:37.911+0000 7f78190a1640 1 -- 172.21.15.64:0/1169180595 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7814143130 con 0x7f78140b4d60 2024-03-25T18:18:38.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.229+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78080035f0 con 0x7f78140b4d60 2024-03-25T18:18:38.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.229+0000 7f77fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:38.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.229+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f78080039c0 con 0x7f78140b4d60 2024-03-25T18:18:38.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.229+0000 7f77fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:38.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.229+0000 7f77fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:38.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.229+0000 7f77fb7fe640 10 monclient: dump: 2024-03-25T18:18:38.231 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:38.231 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:38.231 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:38.231 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:38.231 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:38.231 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:38.231 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:38.231 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:38.231 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:38.231 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:38.231 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.229+0000 7f781159a640 4 set_mon_vals no callback set 2024-03-25T18:18:38.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.230+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f7808003be0 con 0x7f78140b4d60 2024-03-25T18:18:38.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.230+0000 7f7812d9d640 10 monclient: get_auth_request con 0x7f780006e130 auth_method 0 2024-03-25T18:18:38.232 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.230+0000 7f77fb7fe640 10 client.4631.objecter ms_handle_connect 0x7f780006e130 2024-03-25T18:18:38.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.238+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f7808054750 con 0x7f78140b4d60 2024-03-25T18:18:38.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.239+0000 7f77fb7fe640 10 client.4631.objecter ms_dispatch 0x7f7814000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:38.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.239+0000 7f77fb7fe640 3 client.4631.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:38.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.239+0000 7f77fb7fe640 3 client.4631.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:38.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.239+0000 7f77fb7fe640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-25T18:18:38.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.239+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7808050260 con 0x7f78140b4d60 2024-03-25T18:18:38.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.239+0000 7f77fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:38.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.239+0000 7f77fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:38.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.441+0000 7f78190a1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:38.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.441+0000 7f78190a1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:38.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.441+0000 7f78190a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:18:38.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.441+0000 7f78190a1640 1 -- 172.21.15.64:0/1169180595 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78141401f0 con 0x7f78140b4d60 2024-03-25T18:18:38.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.522+0000 7f77fb7fe640 1 -- 172.21.15.64:0/1169180595 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f7808052190 con 0x7f78140b4d60 2024-03-25T18:18:38.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.522+0000 7f77fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:38.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.522+0000 7f77fb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:38.524 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:18:38.524 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:38.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.524+0000 7f78190a1640 10 librados: watch_flush enter 2024-03-25T18:18:38.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.524+0000 7f78190a1640 10 librados: watch_flush exit 2024-03-25T18:18:38.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.524+0000 7f78190a1640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-25T18:18:38.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.524+0000 7f78190a1640 10 client.4631.objecter successfully canceled tick 2024-03-25T18:18:38.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.525+0000 7f78190a1640 1 -- 172.21.15.64:0/1169180595 >> v1:172.21.15.64:6800/2915543638 conn(0x7f780006e130 legacy=0x7f78000705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:38.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.525+0000 7f78190a1640 10 monclient: shutdown 2024-03-25T18:18:38.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.525+0000 7f78190a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:38.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.525+0000 7f78190a1640 1 -- 172.21.15.64:0/1169180595 >> v1:172.21.15.64:6790/0 conn(0x7f78140b4d60 legacy=0x7f7814138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:38.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.525+0000 7f78190a1640 1 -- 172.21.15.64:0/1169180595 shutdown_connections 2024-03-25T18:18:38.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.525+0000 7f78190a1640 1 -- 172.21.15.64:0/1169180595 >> 172.21.15.64:0/1169180595 conn(0x7f78140617a0 msgr2=0x7f7814061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:38.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.525+0000 7f78190a1640 1 -- 172.21.15.64:0/1169180595 shutdown_connections 2024-03-25T18:18:38.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.525+0000 7f78190a1640 1 -- 172.21.15.64:0/1169180595 wait complete. 2024-03-25T18:18:38.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:38.525+0000 7f78190a1640 1 librados: shutdown 2024-03-25T18:18:38.537 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:43.313 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:26.426 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:43.320 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:26.433 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:43.359 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:26.472 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:43.369 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:09:26.448 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:43.377 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:26.489 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:43.381 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:09:26.459 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:43.391 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:26.503 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:43.411 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:09:26.489 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:43.423 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:26.535 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:45.344 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:45.463 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.463+0000 7f3cc0aff640 10 monclient: get_monmap_and_config 2024-03-25T18:18:45.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.463+0000 7f3cc0aff640 10 monclient: build_initial_monmap 2024-03-25T18:18:45.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.463+0000 7f3cc0aff640 10 monclient: monmap: 2024-03-25T18:18:45.464 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:18:45.464 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:45.464 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:18:45.463929+0000 2024-03-25T18:18:45.464 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:18:45.463929+0000 2024-03-25T18:18:45.464 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:45.464 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:45.464 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:18:45.464 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:18:45.464 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:18:45.464 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:45.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.463+0000 7f3cc0aff640 1 Processor -- start 2024-03-25T18:18:45.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.464+0000 7f3cc0aff640 1 -- start start 2024-03-25T18:18:45.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.464+0000 7f3cc0aff640 10 monclient: init 2024-03-25T18:18:45.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.464+0000 7f3cc0aff640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:45.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.464+0000 7f3cc0aff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:45.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.464+0000 7f3cc0aff640 10 monclient(hunting): picked mon.noname-b con 0x7f3cbc0a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:18:45.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.464+0000 7f3cc0aff640 10 monclient(hunting): picked mon.noname-a con 0x7f3cbc13f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:18:45.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.464+0000 7f3cc0aff640 10 monclient(hunting): picked mon.noname-c con 0x7f3cbc1430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:18:45.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.464+0000 7f3cc0aff640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cbc142520 con 0x7f3cbc13f710 2024-03-25T18:18:45.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.464+0000 7f3cc0aff640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cbc0b28a0 con 0x7f3cbc1430c0 2024-03-25T18:18:45.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.464+0000 7f3cc0aff640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cbc0b2a30 con 0x7f3cbc0a98f0 2024-03-25T18:18:45.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.464+0000 7f3cc0aff640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:45.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.464+0000 7f3cc0aff640 10 monclient(hunting): authenticate will time out at 7511.703286s 2024-03-25T18:18:45.468 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.467+0000 7f3cb3fff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f3cbc13f710 0x7f3cbc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:33036/0 (socket says 172.21.15.64:33036) 2024-03-25T18:18:45.468 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.467+0000 7f3cb3fff640 1 -- 172.21.15.64:0/448953903 learned_addr learned my addr 172.21.15.64:0/448953903 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:18:45.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.490+0000 7f3cb9d9b640 1 -- 172.21.15.64:0/448953903 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309346046 0 0) 0x7f3cbc142520 con 0x7f3cbc13f710 2024-03-25T18:18:45.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.490+0000 7f3cb9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:45.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.490+0000 7f3cb9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:45.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.490+0000 7f3cb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:45.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.490+0000 7f3cb9d9b640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:18:45.491 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.490+0000 7f3cb9d9b640 1 -- 172.21.15.64:0/448953903 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ca8003520 con 0x7f3cbc13f710 2024-03-25T18:18:45.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.491+0000 7f3cb9d9b640 1 -- 172.21.15.64:0/448953903 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308218549 0 0) 0x7f3cbc0b28a0 con 0x7f3cbc1430c0 2024-03-25T18:18:45.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.491+0000 7f3cb9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:45.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.491+0000 7f3cb9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:45.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.491+0000 7f3cb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:45.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.491+0000 7f3cb9d9b640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:18:45.492 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.491+0000 7f3cb9d9b640 1 -- 172.21.15.64:0/448953903 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cbc142520 con 0x7f3cbc1430c0 2024-03-25T18:18:45.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.495+0000 7f3cb9d9b640 1 -- 172.21.15.64:0/448953903 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450162793 0 0) 0x7f3cbc0b2a30 con 0x7f3cbc0a98f0 2024-03-25T18:18:45.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.495+0000 7f3cb9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:45.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.495+0000 7f3cb9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:45.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.495+0000 7f3cb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:45.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.495+0000 7f3cb9d9b640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:18:45.497 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.495+0000 7f3cb9d9b640 1 -- 172.21.15.64:0/448953903 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cbc0b28a0 con 0x7f3cbc0a98f0 2024-03-25T18:18:45.506 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.505+0000 7f3cb9d9b640 1 -- 172.21.15.64:0/448953903 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1865789508 0 0) 0x7f3cbc142520 con 0x7f3cbc1430c0 2024-03-25T18:18:45.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.505+0000 7f3cb9d9b640 1 -- 172.21.15.64:0/448953903 >> v1:172.21.15.64:6790/0 conn(0x7f3cbc0a98f0 legacy=0x7f3cbc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.505+0000 7f3cb9d9b640 1 -- 172.21.15.64:0/448953903 >> v1:172.21.15.64:6789/0 conn(0x7f3cbc13f710 legacy=0x7f3cbc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.505+0000 7f3cb9d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:18:45.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.505+0000 7f3cb9d9b640 1 monclient: found mon.noname-c 2024-03-25T18:18:45.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.505+0000 7f3cb9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:45.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.505+0000 7f3cb9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:18:45.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.505+0000 7f3cb9d9b640 1 -- 172.21.15.64:0/448953903 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cbc0b44c0 con 0x7f3cbc1430c0 2024-03-25T18:18:45.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.505+0000 7f3cb9d9b640 10 monclient: _finish_auth 0 2024-03-25T18:18:45.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.505+0000 7f3cb9d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:18:45.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.505+0000 7f3cb9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:45.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.505+0000 7f3cb9d9b640 1 -- 172.21.15.64:0/448953903 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f3cb40019f0 con 0x7f3cbc1430c0 2024-03-25T18:18:45.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.505+0000 7f3cb9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:45.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.505+0000 7f3cb9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:45.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.505+0000 7f3cb9d9b640 10 monclient: dump: 2024-03-25T18:18:45.507 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:45.508 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:45.508 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:45.508 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:45.508 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:45.508 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:45.508 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:45.508 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:45.508 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:45.508 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:45.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.505+0000 7f3cc0aff640 5 monclient: authenticate success, global_id 4639 2024-03-25T18:18:45.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.505+0000 7f3cc0aff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:45.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.520+0000 7f3cb9d9b640 1 -- 172.21.15.64:0/448953903 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3cb4001b50 con 0x7f3cbc1430c0 2024-03-25T18:18:45.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.520+0000 7f3cb9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:45.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.520+0000 7f3cb9d9b640 1 -- 172.21.15.64:0/448953903 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f3cb4001f60 con 0x7f3cbc1430c0 2024-03-25T18:18:45.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.520+0000 7f3cb9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:45.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.520+0000 7f3cb9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:45.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.520+0000 7f3cb9d9b640 10 monclient: dump: 2024-03-25T18:18:45.521 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:45.521 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:45.521 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:45.521 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:45.521 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:45.521 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:45.521 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:45.521 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:45.521 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:45.521 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:45.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.520+0000 7f3cc0aff640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:45.522 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.520+0000 7f3cc0aff640 4 set_mon_vals no callback set 2024-03-25T18:18:45.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.520+0000 7f3cc0aff640 10 monclient: shutdown 2024-03-25T18:18:45.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.520+0000 7f3cc0aff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:45.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.520+0000 7f3cc0aff640 1 -- 172.21.15.64:0/448953903 >> v1:172.21.15.86:6789/0 conn(0x7f3cbc1430c0 legacy=0x7f3cbc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.521+0000 7f3cc0aff640 1 -- 172.21.15.64:0/448953903 shutdown_connections 2024-03-25T18:18:45.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.521+0000 7f3cc0aff640 1 -- 172.21.15.64:0/448953903 >> 172.21.15.64:0/448953903 conn(0x7f3cbc0b7ea0 msgr2=0x7f3cbc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:45.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.521+0000 7f3cc0aff640 1 -- 172.21.15.64:0/448953903 shutdown_connections 2024-03-25T18:18:45.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.521+0000 7f3cc0aff640 1 -- 172.21.15.64:0/448953903 wait complete. 2024-03-25T18:18:45.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.522+0000 7f3cc0aff640 10 monclient: build_initial_monmap 2024-03-25T18:18:45.523 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.522+0000 7f3cc0aff640 10 monclient: monmap: 2024-03-25T18:18:45.523 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:18:45.523 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:45.524 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:18:45.524 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:18:45.524 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:45.524 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:18:45.524 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:18:45.524 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:18:45.524 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:18:45.524 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:45.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.522+0000 7f3cc0aff640 1 librados: starting msgr at 2024-03-25T18:18:45.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.522+0000 7f3cc0aff640 1 librados: starting objecter 2024-03-25T18:18:45.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.522+0000 7f3cc0aff640 1 Processor -- start 2024-03-25T18:18:45.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.522+0000 7f3cc0aff640 1 -- start start 2024-03-25T18:18:45.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.522+0000 7f3cc0aff640 1 librados: setting wanted keys 2024-03-25T18:18:45.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.522+0000 7f3cc0aff640 1 librados: calling monclient init 2024-03-25T18:18:45.524 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.522+0000 7f3cc0aff640 10 monclient: init 2024-03-25T18:18:45.525 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.523+0000 7f3cc0aff640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:45.525 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.523+0000 7f3cc0aff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:45.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.523+0000 7f3cc0aff640 10 monclient(hunting): picked mon.noname-a con 0x7f3cbc0a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:18:45.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.523+0000 7f3cc0aff640 10 monclient(hunting): picked mon.noname-c con 0x7f3cbc13f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:18:45.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.523+0000 7f3cc0aff640 10 monclient(hunting): picked mon.noname-b con 0x7f3cbc1430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:18:45.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.523+0000 7f3cc0aff640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cbc1388a0 con 0x7f3cbc0a98f0 2024-03-25T18:18:45.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.523+0000 7f3cc0aff640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cbc1349e0 con 0x7f3cbc1430c0 2024-03-25T18:18:45.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.523+0000 7f3cc0aff640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cbc138d30 con 0x7f3cbc13f710 2024-03-25T18:18:45.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.523+0000 7f3cc0aff640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:45.526 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.523+0000 7f3cc0aff640 10 monclient(hunting): authenticate will time out at 7511.761874s 2024-03-25T18:18:45.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.528+0000 7f3cbad9d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3cbc1430c0 0x7f3cbc1342d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:51084/0 (socket says 172.21.15.64:51084) 2024-03-25T18:18:45.529 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.528+0000 7f3cbad9d640 1 -- 172.21.15.64:0/2037013199 learned_addr learned my addr 172.21.15.64:0/2037013199 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:18:45.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.532+0000 7f3cb2ffd640 10 client.?.objecter ms_handle_connect 0x7f3cbc1430c0 2024-03-25T18:18:45.533 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.532+0000 7f3cb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:45.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.545+0000 7f3cb2ffd640 10 client.?.objecter ms_handle_connect 0x7f3cbc13f710 2024-03-25T18:18:45.546 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.545+0000 7f3cb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:45.549 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.548+0000 7f3cb2ffd640 10 client.?.objecter ms_handle_connect 0x7f3cbc0a98f0 2024-03-25T18:18:45.549 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.548+0000 7f3cb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:45.549 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.548+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825714991 0 0) 0x7f3cbc1349e0 con 0x7f3cbc1430c0 2024-03-25T18:18:45.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.549+0000 7f3cb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:45.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.549+0000 7f3cb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:45.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.549+0000 7f3cb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:45.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.549+0000 7f3cb2ffd640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:18:45.550 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.549+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c9c004eb0 con 0x7f3cbc1430c0 2024-03-25T18:18:45.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.565+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183695030 0 0) 0x7f3cbc138d30 con 0x7f3cbc13f710 2024-03-25T18:18:45.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.565+0000 7f3cb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:45.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.565+0000 7f3cb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:45.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.565+0000 7f3cb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:45.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.565+0000 7f3cb2ffd640 10 monclient(hunting): my global_id is 4637 2024-03-25T18:18:45.566 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.565+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cbc1349e0 con 0x7f3cbc13f710 2024-03-25T18:18:45.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.574+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870923289 0 0) 0x7f3cbc1388a0 con 0x7f3cbc0a98f0 2024-03-25T18:18:45.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.574+0000 7f3cb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:45.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.574+0000 7f3cb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:45.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.574+0000 7f3cb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:45.575 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.574+0000 7f3cb2ffd640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:18:45.576 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.574+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cbc138d30 con 0x7f3cbc0a98f0 2024-03-25T18:18:45.577 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.576+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2450630220 0 0) 0x7f3c9c004eb0 con 0x7f3cbc1430c0 2024-03-25T18:18:45.577 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.576+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cbc1388a0 con 0x7f3cbc1430c0 2024-03-25T18:18:45.577 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.576+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f3cb4001fa0 con 0x7f3cbc1430c0 2024-03-25T18:18:45.577 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.576+0000 7f3cb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:45.577 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.576+0000 7f3cb2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:45.577 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.576+0000 7f3cb2ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:45.577 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:45.578 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:45.578 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:45.578 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:45.578 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:45.578 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:45.578 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:45.578 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:45.578 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:45.578 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:45.578 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.576+0000 7f3cb2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:45.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.588+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4129470110 0 0) 0x7f3cbc1349e0 con 0x7f3cbc13f710 2024-03-25T18:18:45.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.588+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c9c004eb0 con 0x7f3cbc13f710 2024-03-25T18:18:45.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.588+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f3ca4001950 con 0x7f3cbc13f710 2024-03-25T18:18:45.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.588+0000 7f3cb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:45.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.588+0000 7f3cb2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:45.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.588+0000 7f3cb2ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:45.589 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:45.590 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:45.590 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:45.590 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:45.590 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:45.590 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:45.590 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:45.590 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:45.590 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:45.590 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:45.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.593+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2671526854 0 0) 0x7f3cbc1388a0 con 0x7f3cbc1430c0 2024-03-25T18:18:45.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 >> v1:172.21.15.64:6790/0 conn(0x7f3cbc13f710 legacy=0x7f3cbc133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 >> v1:172.21.15.64:6789/0 conn(0x7f3cbc0a98f0 legacy=0x7f3cbc1da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cb2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:45.595 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cb2ffd640 1 monclient: found mon.b 2024-03-25T18:18:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cb2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:18:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cbc0aba50 con 0x7f3cbc1430c0 2024-03-25T18:18:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cb2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:18:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cb2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cc0aff640 5 monclient: authenticate success, global_id 4642 2024-03-25T18:18:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cc0aff640 10 monclient: _renew_subs 2024-03-25T18:18:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cc0aff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:18:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cc0aff640 1 -- 172.21.15.64:0/2037013199 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cbc0b1770 con 0x7f3cbc1430c0 2024-03-25T18:18:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cc0aff640 10 client.4642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cc0aff640 10 monclient: _renew_subs 2024-03-25T18:18:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cc0aff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:18:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cc0aff640 1 -- 172.21.15.64:0/2037013199 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cbc137ba0 con 0x7f3cbc1430c0 2024-03-25T18:18:45.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.594+0000 7f3cc0aff640 1 librados: init done 2024-03-25T18:18:45.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.595+0000 7f3cc0aff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:45.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.595+0000 7f3cc0aff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:45.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.595+0000 7f3cc0aff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:18:45.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.595+0000 7f3cc0aff640 1 -- 172.21.15.64:0/2037013199 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cbc0627b0 con 0x7f3cbc1430c0 2024-03-25T18:18:45.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.896+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3cb4003c20 con 0x7f3cbc1430c0 2024-03-25T18:18:45.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.896+0000 7f3cb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:45.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.896+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f3cb4003f70 con 0x7f3cbc1430c0 2024-03-25T18:18:45.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.896+0000 7f3cb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:45.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.896+0000 7f3cb959a640 4 set_mon_vals no callback set 2024-03-25T18:18:45.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.896+0000 7f3cb2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:45.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.896+0000 7f3cb2ffd640 10 monclient: dump: 2024-03-25T18:18:45.898 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:45.898 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:45.898 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:45.898 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:45.898 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:45.898 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:45.898 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:45.898 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:45.898 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:45.898 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:45.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.897+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f3cb4004190 con 0x7f3cbc1430c0 2024-03-25T18:18:45.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.897+0000 7f3cba59c640 10 monclient: get_auth_request con 0x7f3c9c06df70 auth_method 0 2024-03-25T18:18:45.899 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.897+0000 7f3cb2ffd640 10 client.4642.objecter ms_handle_connect 0x7f3c9c06df70 2024-03-25T18:18:45.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.903+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f3cb4054b60 con 0x7f3cbc1430c0 2024-03-25T18:18:45.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.903+0000 7f3cb2ffd640 10 client.4642.objecter ms_dispatch 0x7f3cbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:45.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.903+0000 7f3cb2ffd640 3 client.4642.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:45.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.903+0000 7f3cb2ffd640 3 client.4642.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:45.904 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.903+0000 7f3cb2ffd640 20 client.4642.objecter dump_active .. 0 homeless 2024-03-25T18:18:45.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.903+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3cb4050670 con 0x7f3cbc1430c0 2024-03-25T18:18:45.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.903+0000 7f3cb2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:45.905 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:45.903+0000 7f3cb2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:46.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.148+0000 7f3cc0aff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:46.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.148+0000 7f3cc0aff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:46.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.148+0000 7f3cc0aff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:18:46.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.148+0000 7f3cc0aff640 1 -- 172.21.15.64:0/2037013199 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3cbc138a50 con 0x7f3cbc1430c0 2024-03-25T18:18:46.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.282+0000 7f3cb2ffd640 1 -- 172.21.15.64:0/2037013199 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f3cbc138a50 con 0x7f3cbc1430c0 2024-03-25T18:18:46.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.282+0000 7f3cb2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:46.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.282+0000 7f3cb2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:46.283 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:18:46.283 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:46.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.284+0000 7f3cc0aff640 10 librados: watch_flush enter 2024-03-25T18:18:46.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.284+0000 7f3cc0aff640 10 librados: watch_flush exit 2024-03-25T18:18:46.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.284+0000 7f3cc0aff640 20 client.4642.objecter shutdown clearing up homeless session... 2024-03-25T18:18:46.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.284+0000 7f3cc0aff640 10 client.4642.objecter successfully canceled tick 2024-03-25T18:18:46.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.284+0000 7f3cc0aff640 1 -- 172.21.15.64:0/2037013199 >> v1:172.21.15.64:6800/2915543638 conn(0x7f3c9c06df70 legacy=0x7f3c9c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:46.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.284+0000 7f3cc0aff640 10 monclient: shutdown 2024-03-25T18:18:46.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.284+0000 7f3cc0aff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:46.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.284+0000 7f3cc0aff640 1 -- 172.21.15.64:0/2037013199 >> v1:172.21.15.86:6789/0 conn(0x7f3cbc1430c0 legacy=0x7f3cbc1342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:46.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.284+0000 7f3cc0aff640 1 -- 172.21.15.64:0/2037013199 shutdown_connections 2024-03-25T18:18:46.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.284+0000 7f3cc0aff640 1 -- 172.21.15.64:0/2037013199 >> 172.21.15.64:0/2037013199 conn(0x7f3cbc0b7ea0 msgr2=0x7f3cbc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:46.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.284+0000 7f3cc0aff640 1 -- 172.21.15.64:0/2037013199 shutdown_connections 2024-03-25T18:18:46.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.284+0000 7f3cc0aff640 1 -- 172.21.15.64:0/2037013199 wait complete. 2024-03-25T18:18:46.286 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:46.284+0000 7f3cc0aff640 1 librados: shutdown 2024-03-25T18:18:46.297 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:53.102 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:53.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.217+0000 7f521119a640 10 monclient: get_monmap_and_config 2024-03-25T18:18:53.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.217+0000 7f521119a640 10 monclient: build_initial_monmap 2024-03-25T18:18:53.218 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.217+0000 7f521119a640 10 monclient: monmap: 2024-03-25T18:18:53.218 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:18:53.218 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:53.218 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:18:53.218277+0000 2024-03-25T18:18:53.218 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:18:53.218277+0000 2024-03-25T18:18:53.218 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:53.218 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:53.219 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:18:53.219 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:18:53.219 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:18:53.219 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:53.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.218+0000 7f521119a640 1 Processor -- start 2024-03-25T18:18:53.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.218+0000 7f521119a640 1 -- start start 2024-03-25T18:18:53.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.218+0000 7f521119a640 10 monclient: init 2024-03-25T18:18:53.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.218+0000 7f521119a640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:53.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.218+0000 7f521119a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:53.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.218+0000 7f521119a640 10 monclient(hunting): picked mon.noname-b con 0x7f520c0a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:18:53.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.218+0000 7f521119a640 10 monclient(hunting): picked mon.noname-a con 0x7f520c13f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:18:53.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.218+0000 7f521119a640 10 monclient(hunting): picked mon.noname-c con 0x7f520c1430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:18:53.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.218+0000 7f521119a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f520c142520 con 0x7f520c13f710 2024-03-25T18:18:53.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.219+0000 7f521119a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f520c0b28a0 con 0x7f520c1430c0 2024-03-25T18:18:53.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.219+0000 7f521119a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f520c0b2a30 con 0x7f520c0a98f0 2024-03-25T18:18:53.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.219+0000 7f521119a640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:53.221 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.219+0000 7f521119a640 10 monclient(hunting): authenticate will time out at 7519.457553s 2024-03-25T18:18:53.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.228+0000 7f520b59e640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f520c1430c0 0x7f520c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:58856/0 (socket says 172.21.15.64:58856) 2024-03-25T18:18:53.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.228+0000 7f520b59e640 1 -- 172.21.15.64:0/1756058689 learned_addr learned my addr 172.21.15.64:0/1756058689 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:18:53.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.245+0000 7f5209d9b640 1 -- 172.21.15.64:0/1756058689 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366769545 0 0) 0x7f520c0b28a0 con 0x7f520c1430c0 2024-03-25T18:18:53.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.246+0000 7f5209d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:53.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.246+0000 7f5209d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:53.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.246+0000 7f5209d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:53.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.246+0000 7f5209d9b640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:18:53.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.246+0000 7f5209d9b640 1 -- 172.21.15.64:0/1756058689 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51f4003520 con 0x7f520c1430c0 2024-03-25T18:18:53.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.255+0000 7f5209d9b640 1 -- 172.21.15.64:0/1756058689 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394121337 0 0) 0x7f520c0b2a30 con 0x7f520c0a98f0 2024-03-25T18:18:53.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.255+0000 7f5209d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:53.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.255+0000 7f5209d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:53.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.255+0000 7f5209d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:53.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.255+0000 7f5209d9b640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:18:53.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.255+0000 7f5209d9b640 1 -- 172.21.15.64:0/1756058689 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f520c0b28a0 con 0x7f520c0a98f0 2024-03-25T18:18:53.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.256+0000 7f5209d9b640 1 -- 172.21.15.64:0/1756058689 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490214370 0 0) 0x7f520c142520 con 0x7f520c13f710 2024-03-25T18:18:53.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.256+0000 7f5209d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:53.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.256+0000 7f5209d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:53.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.256+0000 7f5209d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:53.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.256+0000 7f5209d9b640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:18:53.257 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.256+0000 7f5209d9b640 1 -- 172.21.15.64:0/1756058689 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f520c0b2a30 con 0x7f520c13f710 2024-03-25T18:18:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.263+0000 7f5209d9b640 1 -- 172.21.15.64:0/1756058689 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 307451925 0 0) 0x7f51f4003520 con 0x7f520c1430c0 2024-03-25T18:18:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.263+0000 7f5209d9b640 1 -- 172.21.15.64:0/1756058689 >> v1:172.21.15.64:6790/0 conn(0x7f520c0a98f0 legacy=0x7f520c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.263+0000 7f5209d9b640 1 -- 172.21.15.64:0/1756058689 >> v1:172.21.15.64:6789/0 conn(0x7f520c13f710 legacy=0x7f520c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.263+0000 7f5209d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:18:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.263+0000 7f5209d9b640 1 monclient: found mon.noname-c 2024-03-25T18:18:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.263+0000 7f5209d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:53.264 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.263+0000 7f5209d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:18:53.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.263+0000 7f5209d9b640 1 -- 172.21.15.64:0/1756058689 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520c0b44c0 con 0x7f520c1430c0 2024-03-25T18:18:53.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.263+0000 7f5209d9b640 10 monclient: _finish_auth 0 2024-03-25T18:18:53.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.263+0000 7f5209d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:18:53.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.263+0000 7f5209d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:53.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.263+0000 7f5209d9b640 1 -- 172.21.15.64:0/1756058689 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f5204001a10 con 0x7f520c1430c0 2024-03-25T18:18:53.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.263+0000 7f521119a640 5 monclient: authenticate success, global_id 4645 2024-03-25T18:18:53.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.263+0000 7f521119a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:53.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.263+0000 7f5209d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:53.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.263+0000 7f5209d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:53.265 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.263+0000 7f5209d9b640 10 monclient: dump: 2024-03-25T18:18:53.265 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:53.265 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:53.265 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:53.265 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:53.265 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:53.265 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:53.265 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:53.266 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:53.266 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:53.266 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:53.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.263+0000 7f521119a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:53.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.269+0000 7f5209d9b640 1 -- 172.21.15.64:0/1756058689 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5204001b70 con 0x7f520c1430c0 2024-03-25T18:18:53.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.269+0000 7f5209d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:53.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.269+0000 7f5209d9b640 1 -- 172.21.15.64:0/1756058689 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f5204001f80 con 0x7f520c1430c0 2024-03-25T18:18:53.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.269+0000 7f5209d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:53.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.269+0000 7f5209d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:53.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.269+0000 7f5209d9b640 10 monclient: dump: 2024-03-25T18:18:53.270 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:53.271 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:53.271 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:53.271 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:53.271 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:53.271 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:53.271 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:53.271 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:53.271 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:53.271 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:53.271 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.269+0000 7f521119a640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:53.271 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.269+0000 7f521119a640 4 set_mon_vals no callback set 2024-03-25T18:18:53.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.270+0000 7f521119a640 10 monclient: shutdown 2024-03-25T18:18:53.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.270+0000 7f521119a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:53.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.270+0000 7f521119a640 1 -- 172.21.15.64:0/1756058689 >> v1:172.21.15.86:6789/0 conn(0x7f520c1430c0 legacy=0x7f520c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.270+0000 7f521119a640 1 -- 172.21.15.64:0/1756058689 shutdown_connections 2024-03-25T18:18:53.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.270+0000 7f521119a640 1 -- 172.21.15.64:0/1756058689 >> 172.21.15.64:0/1756058689 conn(0x7f520c0b7ea0 msgr2=0x7f520c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:53.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.270+0000 7f521119a640 1 -- 172.21.15.64:0/1756058689 shutdown_connections 2024-03-25T18:18:53.272 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.270+0000 7f521119a640 1 -- 172.21.15.64:0/1756058689 wait complete. 2024-03-25T18:18:53.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.271+0000 7f521119a640 10 monclient: build_initial_monmap 2024-03-25T18:18:53.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.271+0000 7f521119a640 10 monclient: monmap: 2024-03-25T18:18:53.273 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:18:53.273 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:53.273 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:18:53.273 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:18:53.273 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:53.273 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:18:53.274 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:18:53.274 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:18:53.274 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:18:53.274 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:53.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.272+0000 7f521119a640 1 librados: starting msgr at 2024-03-25T18:18:53.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.272+0000 7f521119a640 1 librados: starting objecter 2024-03-25T18:18:53.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.272+0000 7f521119a640 1 Processor -- start 2024-03-25T18:18:53.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.272+0000 7f521119a640 1 -- start start 2024-03-25T18:18:53.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.272+0000 7f521119a640 1 librados: setting wanted keys 2024-03-25T18:18:53.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.272+0000 7f521119a640 1 librados: calling monclient init 2024-03-25T18:18:53.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.272+0000 7f521119a640 10 monclient: init 2024-03-25T18:18:53.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.273+0000 7f521119a640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:53.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.273+0000 7f521119a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:53.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.273+0000 7f521119a640 10 monclient(hunting): picked mon.noname-a con 0x7f520c0a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:18:53.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.273+0000 7f521119a640 10 monclient(hunting): picked mon.noname-c con 0x7f520c13f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:18:53.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.273+0000 7f521119a640 10 monclient(hunting): picked mon.noname-b con 0x7f520c1430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:18:53.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.273+0000 7f521119a640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f520c1dacc0 con 0x7f520c0a98f0 2024-03-25T18:18:53.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.273+0000 7f521119a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f520c139990 con 0x7f520c1430c0 2024-03-25T18:18:53.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.273+0000 7f521119a640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f520c1db860 con 0x7f520c13f710 2024-03-25T18:18:53.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.273+0000 7f521119a640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:53.276 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.273+0000 7f521119a640 10 monclient(hunting): authenticate will time out at 7519.511823s 2024-03-25T18:18:53.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.285+0000 7f520ad9d640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f520c0a98f0 0x7f520c137a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:39634/0 (socket says 172.21.15.64:39634) 2024-03-25T18:18:53.285 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.285+0000 7f520ad9d640 1 -- 172.21.15.64:0/3179686859 learned_addr learned my addr 172.21.15.64:0/3179686859 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:18:53.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.287+0000 7f51f37fe640 10 client.?.objecter ms_handle_connect 0x7f520c0a98f0 2024-03-25T18:18:53.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.287+0000 7f51f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:53.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.294+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145452273 0 0) 0x7f520c1dacc0 con 0x7f520c0a98f0 2024-03-25T18:18:53.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.294+0000 7f51f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:53.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.294+0000 7f51f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:53.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.294+0000 7f51f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:53.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.294+0000 7f51f37fe640 10 monclient(hunting): my global_id is 4644 2024-03-25T18:18:53.295 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.294+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51f8004eb0 con 0x7f520c0a98f0 2024-03-25T18:18:53.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.297+0000 7f51f37fe640 10 client.?.objecter ms_handle_connect 0x7f520c1430c0 2024-03-25T18:18:53.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.297+0000 7f51f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:53.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.303+0000 7f51f37fe640 10 client.?.objecter ms_handle_connect 0x7f520c13f710 2024-03-25T18:18:53.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.303+0000 7f51f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:53.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.308+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43417781 0 0) 0x7f520c139990 con 0x7f520c1430c0 2024-03-25T18:18:53.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.308+0000 7f51f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:53.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.308+0000 7f51f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:53.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.308+0000 7f51f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:53.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.308+0000 7f51f37fe640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:18:53.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.308+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f520c1dacc0 con 0x7f520c1430c0 2024-03-25T18:18:53.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.313+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673823653 0 0) 0x7f520c1db860 con 0x7f520c13f710 2024-03-25T18:18:53.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.313+0000 7f51f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:53.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.313+0000 7f51f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:53.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.313+0000 7f51f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:53.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.313+0000 7f51f37fe640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:18:53.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.313+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f520c139990 con 0x7f520c13f710 2024-03-25T18:18:53.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.315+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1114847927 0 0) 0x7f51f8004eb0 con 0x7f520c0a98f0 2024-03-25T18:18:53.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.315+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f520c1db860 con 0x7f520c0a98f0 2024-03-25T18:18:53.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.315+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f51fc001a30 con 0x7f520c0a98f0 2024-03-25T18:18:53.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.315+0000 7f51f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:53.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.315+0000 7f51f37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:53.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.315+0000 7f51f37fe640 10 monclient(hunting): dump: 2024-03-25T18:18:53.316 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:53.316 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:53.316 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:53.316 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:53.316 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:53.316 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:53.316 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:53.316 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:53.316 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:53.317 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:53.317 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.315+0000 7f51f37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:53.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 210042170 0 0) 0x7f520c1db860 con 0x7f520c0a98f0 2024-03-25T18:18:53.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 >> v1:172.21.15.64:6790/0 conn(0x7f520c13f710 legacy=0x7f520c1db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.324 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 >> v1:172.21.15.86:6789/0 conn(0x7f520c1430c0 legacy=0x7f520c1e2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f51f37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:53.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f51f37fe640 1 monclient: found mon.a 2024-03-25T18:18:53.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f51f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:53.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f51f37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:18:53.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520c135040 con 0x7f520c0a98f0 2024-03-25T18:18:53.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f51f37fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:53.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f51f37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:53.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f51f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:53.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f521119a640 5 monclient: authenticate success, global_id 4644 2024-03-25T18:18:53.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f521119a640 10 monclient: _renew_subs 2024-03-25T18:18:53.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f521119a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:18:53.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f521119a640 1 -- 172.21.15.64:0/3179686859 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f520c1352f0 con 0x7f520c0a98f0 2024-03-25T18:18:53.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f521119a640 10 client.4644.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:53.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f521119a640 10 monclient: _renew_subs 2024-03-25T18:18:53.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f521119a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:18:53.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f521119a640 1 -- 172.21.15.64:0/3179686859 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f520c138a70 con 0x7f520c0a98f0 2024-03-25T18:18:53.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.323+0000 7f521119a640 1 librados: init done 2024-03-25T18:18:53.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.324+0000 7f521119a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:53.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.324+0000 7f521119a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:53.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.324+0000 7f521119a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:18:53.326 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.324+0000 7f521119a640 1 -- 172.21.15.64:0/3179686859 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f520c138d20 con 0x7f520c0a98f0 2024-03-25T18:18:53.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.535+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51fc001cf0 con 0x7f520c0a98f0 2024-03-25T18:18:53.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.535+0000 7f51f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:53.536 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.535+0000 7f520959a640 4 set_mon_vals no callback set 2024-03-25T18:18:53.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.626+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f51fc003a60 con 0x7f520c0a98f0 2024-03-25T18:18:53.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.626+0000 7f51f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:53.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.626+0000 7f51f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:53.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.626+0000 7f51f37fe640 10 monclient: dump: 2024-03-25T18:18:53.627 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:18:53.627 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:18:53.627 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:53.627 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:18:53.627 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:53.627 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:18:53.627 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:18:53.628 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:18:53.628 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:18:53.628 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:18:53.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.627+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f51fc003c80 con 0x7f520c0a98f0 2024-03-25T18:18:53.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.627+0000 7f520a59c640 10 monclient: get_auth_request con 0x7f51f806dd50 auth_method 0 2024-03-25T18:18:53.628 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.628+0000 7f51f37fe640 10 client.4644.objecter ms_handle_connect 0x7f51f806dd50 2024-03-25T18:18:53.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.635+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f51fc054960 con 0x7f520c0a98f0 2024-03-25T18:18:53.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.635+0000 7f51f37fe640 10 client.4644.objecter ms_dispatch 0x7f520c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:53.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.635+0000 7f51f37fe640 3 client.4644.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:53.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.635+0000 7f51f37fe640 3 client.4644.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:53.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.635+0000 7f51f37fe640 20 client.4644.objecter dump_active .. 0 homeless 2024-03-25T18:18:53.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.635+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f51fc050470 con 0x7f520c0a98f0 2024-03-25T18:18:53.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.635+0000 7f51f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:53.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.635+0000 7f51f37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:53.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.897+0000 7f521119a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:53.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.897+0000 7f521119a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:53.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.897+0000 7f521119a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:18:53.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.897+0000 7f521119a640 1 -- 172.21.15.64:0/3179686859 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f520c1daa20 con 0x7f520c0a98f0 2024-03-25T18:18:53.923 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.923+0000 7f51f37fe640 1 -- 172.21.15.64:0/3179686859 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f51fc0523a0 con 0x7f520c0a98f0 2024-03-25T18:18:53.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.923+0000 7f51f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:53.924 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.923+0000 7f51f37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:53.924 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:18:53.924 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:53.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.925+0000 7f521119a640 10 librados: watch_flush enter 2024-03-25T18:18:53.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.925+0000 7f521119a640 10 librados: watch_flush exit 2024-03-25T18:18:53.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.925+0000 7f521119a640 20 client.4644.objecter shutdown clearing up homeless session... 2024-03-25T18:18:53.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.925+0000 7f521119a640 10 client.4644.objecter successfully canceled tick 2024-03-25T18:18:53.926 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.925+0000 7f521119a640 1 -- 172.21.15.64:0/3179686859 >> v1:172.21.15.64:6800/2915543638 conn(0x7f51f806dd50 legacy=0x7f51f8070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.925+0000 7f521119a640 10 monclient: shutdown 2024-03-25T18:18:53.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.925+0000 7f521119a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:53.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.925+0000 7f521119a640 1 -- 172.21.15.64:0/3179686859 >> v1:172.21.15.64:6789/0 conn(0x7f520c0a98f0 legacy=0x7f520c137a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.925+0000 7f521119a640 1 -- 172.21.15.64:0/3179686859 shutdown_connections 2024-03-25T18:18:53.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.925+0000 7f521119a640 1 -- 172.21.15.64:0/3179686859 >> 172.21.15.64:0/3179686859 conn(0x7f520c0b7ea0 msgr2=0x7f520c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:53.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.925+0000 7f521119a640 1 -- 172.21.15.64:0/3179686859 shutdown_connections 2024-03-25T18:18:53.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.925+0000 7f521119a640 1 -- 172.21.15.64:0/3179686859 wait complete. 2024-03-25T18:18:53.927 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:18:53.925+0000 7f521119a640 1 librados: shutdown 2024-03-25T18:18:53.938 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:00.743 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:00.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.853+0000 7f03383e8640 10 monclient: get_monmap_and_config 2024-03-25T18:19:00.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.853+0000 7f03383e8640 10 monclient: build_initial_monmap 2024-03-25T18:19:00.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.853+0000 7f03383e8640 10 monclient: monmap: 2024-03-25T18:19:00.854 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:19:00.854 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:00.854 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:19:00.854163+0000 2024-03-25T18:19:00.854 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:19:00.854163+0000 2024-03-25T18:19:00.854 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:00.855 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:00.855 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:19:00.855 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:19:00.855 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:19:00.855 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:00.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.853+0000 7f03383e8640 1 Processor -- start 2024-03-25T18:19:00.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.853+0000 7f03383e8640 1 -- start start 2024-03-25T18:19:00.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.853+0000 7f03383e8640 10 monclient: init 2024-03-25T18:19:00.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.854+0000 7f03383e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:00.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.854+0000 7f03383e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:00.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.854+0000 7f03383e8640 10 monclient(hunting): picked mon.noname-c con 0x7f03300a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:19:00.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.854+0000 7f03383e8640 10 monclient(hunting): picked mon.noname-a con 0x7f033013f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:19:00.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.854+0000 7f03383e8640 10 monclient(hunting): picked mon.noname-b con 0x7f03301430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:19:00.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.854+0000 7f03383e8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0330142520 con 0x7f033013f710 2024-03-25T18:19:00.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.854+0000 7f03383e8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03300b28a0 con 0x7f03300a98f0 2024-03-25T18:19:00.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.854+0000 7f03383e8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03300b2a30 con 0x7f03301430c0 2024-03-25T18:19:00.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.854+0000 7f03383e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:00.856 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.854+0000 7f03383e8640 10 monclient(hunting): authenticate will time out at 7527.093282s 2024-03-25T18:19:00.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.868+0000 7f0336184640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f03300a98f0 0x7f03300a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:38382/0 (socket says 172.21.15.64:38382) 2024-03-25T18:19:00.869 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.868+0000 7f0336184640 1 -- 172.21.15.64:0/1520916597 learned_addr learned my addr 172.21.15.64:0/1520916597 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:19:00.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.878+0000 7f0335182640 1 -- 172.21.15.64:0/1520916597 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423580575 0 0) 0x7f03300b28a0 con 0x7f03300a98f0 2024-03-25T18:19:00.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.878+0000 7f0335182640 10 monclient(hunting): _negotiate 2024-03-25T18:19:00.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.878+0000 7f0335182640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:00.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.878+0000 7f0335182640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:00.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.878+0000 7f0335182640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:19:00.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.879+0000 7f0335182640 1 -- 172.21.15.64:0/1520916597 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f031c003520 con 0x7f03300a98f0 2024-03-25T18:19:00.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.890+0000 7f0335182640 1 -- 172.21.15.64:0/1520916597 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 595590425 0 0) 0x7f031c003520 con 0x7f03300a98f0 2024-03-25T18:19:00.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.890+0000 7f0335182640 1 -- 172.21.15.64:0/1520916597 >> v1:172.21.15.64:6790/0 conn(0x7f03301430c0 legacy=0x7f033014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:00.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.890+0000 7f0335182640 1 -- 172.21.15.64:0/1520916597 >> v1:172.21.15.64:6789/0 conn(0x7f033013f710 legacy=0x7f03301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:00.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.890+0000 7f0335182640 10 monclient: _finish_hunting 0 2024-03-25T18:19:00.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.890+0000 7f0335182640 1 monclient: found mon.noname-c 2024-03-25T18:19:00.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.890+0000 7f0335182640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:00.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.890+0000 7f0335182640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:19:00.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.890+0000 7f0335182640 1 -- 172.21.15.64:0/1520916597 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03300b44c0 con 0x7f03300a98f0 2024-03-25T18:19:00.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.890+0000 7f0335182640 10 monclient: _finish_auth 0 2024-03-25T18:19:00.892 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.890+0000 7f0335182640 10 monclient: _check_auth_tickets 2024-03-25T18:19:00.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.890+0000 7f0335182640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:00.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.890+0000 7f0335182640 1 -- 172.21.15.64:0/1520916597 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f032c001690 con 0x7f03300a98f0 2024-03-25T18:19:00.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.890+0000 7f0335182640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:00.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.890+0000 7f0335182640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:00.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.890+0000 7f0335182640 10 monclient: dump: 2024-03-25T18:19:00.893 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:00.893 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:00.893 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:00.893 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:00.893 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:00.893 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:00.893 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:00.893 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:00.893 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:00.893 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:00.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.890+0000 7f03383e8640 5 monclient: authenticate success, global_id 4651 2024-03-25T18:19:00.893 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.890+0000 7f03383e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:00.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.909+0000 7f0335182640 1 -- 172.21.15.64:0/1520916597 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f032c0017f0 con 0x7f03300a98f0 2024-03-25T18:19:00.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.909+0000 7f0335182640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:00.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.909+0000 7f0335182640 1 -- 172.21.15.64:0/1520916597 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f032c001c00 con 0x7f03300a98f0 2024-03-25T18:19:00.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.909+0000 7f03383e8640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:00.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.909+0000 7f03383e8640 4 set_mon_vals no callback set 2024-03-25T18:19:00.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.909+0000 7f0335182640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:00.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.909+0000 7f0335182640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:00.911 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.909+0000 7f0335182640 10 monclient: dump: 2024-03-25T18:19:00.911 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:00.911 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:00.911 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:00.911 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:00.911 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:00.911 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:00.911 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:00.911 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:00.911 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:00.911 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:00.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.910+0000 7f03383e8640 10 monclient: shutdown 2024-03-25T18:19:00.912 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.910+0000 7f03383e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:00.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.910+0000 7f03383e8640 1 -- 172.21.15.64:0/1520916597 >> v1:172.21.15.86:6789/0 conn(0x7f03300a98f0 legacy=0x7f03300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:00.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.910+0000 7f03383e8640 1 -- 172.21.15.64:0/1520916597 shutdown_connections 2024-03-25T18:19:00.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.910+0000 7f03383e8640 1 -- 172.21.15.64:0/1520916597 >> 172.21.15.64:0/1520916597 conn(0x7f03300b7ea0 msgr2=0x7f03300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:00.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.910+0000 7f03383e8640 1 -- 172.21.15.64:0/1520916597 shutdown_connections 2024-03-25T18:19:00.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.910+0000 7f03383e8640 1 -- 172.21.15.64:0/1520916597 wait complete. 2024-03-25T18:19:00.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.911+0000 7f03383e8640 10 monclient: build_initial_monmap 2024-03-25T18:19:00.913 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.911+0000 7f03383e8640 10 monclient: monmap: 2024-03-25T18:19:00.913 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:19:00.913 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:00.913 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:19:00.913 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:19:00.913 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:00.913 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:19:00.913 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:19:00.913 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:19:00.913 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:19:00.914 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:00.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.911+0000 7f03383e8640 1 librados: starting msgr at 2024-03-25T18:19:00.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.911+0000 7f03383e8640 1 librados: starting objecter 2024-03-25T18:19:00.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.912+0000 7f03383e8640 1 Processor -- start 2024-03-25T18:19:00.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.912+0000 7f03383e8640 1 -- start start 2024-03-25T18:19:00.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.912+0000 7f03383e8640 1 librados: setting wanted keys 2024-03-25T18:19:00.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.912+0000 7f03383e8640 1 librados: calling monclient init 2024-03-25T18:19:00.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.912+0000 7f03383e8640 10 monclient: init 2024-03-25T18:19:00.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.912+0000 7f03383e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:00.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.912+0000 7f03383e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:00.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.912+0000 7f03383e8640 10 monclient(hunting): picked mon.noname-b con 0x7f03300a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:19:00.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.912+0000 7f03383e8640 10 monclient(hunting): picked mon.noname-a con 0x7f033013f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:19:00.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.912+0000 7f03383e8640 10 monclient(hunting): picked mon.noname-c con 0x7f03301430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:19:00.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.912+0000 7f03383e8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03300ab9e0 con 0x7f033013f710 2024-03-25T18:19:00.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.912+0000 7f03383e8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0330138c20 con 0x7f03300a98f0 2024-03-25T18:19:00.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.912+0000 7f03383e8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0330138f00 con 0x7f03301430c0 2024-03-25T18:19:00.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.912+0000 7f03383e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:00.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.912+0000 7f03383e8640 10 monclient(hunting): authenticate will time out at 7527.151354s 2024-03-25T18:19:00.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.927+0000 7f0335983640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f033013f710 0x7f03301d7950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:43024/0 (socket says 172.21.15.64:43024) 2024-03-25T18:19:00.928 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.927+0000 7f0335983640 1 -- 172.21.15.64:0/4287021535 learned_addr learned my addr 172.21.15.64:0/4287021535 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:19:00.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.930+0000 7f0326ffd640 10 client.?.objecter ms_handle_connect 0x7f03300a98f0 2024-03-25T18:19:00.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.930+0000 7f0326ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:00.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.930+0000 7f0326ffd640 10 client.?.objecter ms_handle_connect 0x7f033013f710 2024-03-25T18:19:00.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.930+0000 7f0326ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:00.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.932+0000 7f0326ffd640 10 client.?.objecter ms_handle_connect 0x7f03301430c0 2024-03-25T18:19:00.933 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.932+0000 7f0326ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:00.939 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.938+0000 7f0326ffd640 1 -- 172.21.15.64:0/4287021535 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106603256 0 0) 0x7f0330138f00 con 0x7f03301430c0 2024-03-25T18:19:00.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.938+0000 7f0326ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:00.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.938+0000 7f0326ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:00.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.938+0000 7f0326ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:00.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.938+0000 7f0326ffd640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:19:00.940 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.938+0000 7f0326ffd640 1 -- 172.21.15.64:0/4287021535 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0314004eb0 con 0x7f03301430c0 2024-03-25T18:19:00.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.957+0000 7f0326ffd640 1 -- 172.21.15.64:0/4287021535 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 720158865 0 0) 0x7f0314004eb0 con 0x7f03301430c0 2024-03-25T18:19:00.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.957+0000 7f0326ffd640 1 -- 172.21.15.64:0/4287021535 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0330138f00 con 0x7f03301430c0 2024-03-25T18:19:00.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.957+0000 7f0326ffd640 1 -- 172.21.15.64:0/4287021535 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0328001b20 con 0x7f03301430c0 2024-03-25T18:19:00.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.957+0000 7f0326ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:00.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.957+0000 7f0326ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:00.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.957+0000 7f0326ffd640 10 monclient(hunting): dump: 2024-03-25T18:19:00.959 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:00.959 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:00.959 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:00.959 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:00.959 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:00.959 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:00.959 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:00.960 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:00.960 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:00.960 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:00.960 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:00.957+0000 7f0326ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:01.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f0326ffd640 1 -- 172.21.15.64:0/4287021535 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2655737673 0 0) 0x7f0330138f00 con 0x7f03301430c0 2024-03-25T18:19:01.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f0326ffd640 1 -- 172.21.15.64:0/4287021535 >> v1:172.21.15.86:6789/0 conn(0x7f03300a98f0 legacy=0x7f0330137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f0326ffd640 1 -- 172.21.15.64:0/4287021535 >> v1:172.21.15.64:6789/0 conn(0x7f033013f710 legacy=0x7f03301d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.038 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f0326ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:01.038 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f0326ffd640 1 monclient: found mon.c 2024-03-25T18:19:01.038 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f0326ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:01.038 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f0326ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:19:01.038 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f0326ffd640 1 -- 172.21.15.64:0/4287021535 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033014e7a0 con 0x7f03301430c0 2024-03-25T18:19:01.038 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f0326ffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:01.038 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f0326ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:01.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f0326ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:01.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f03383e8640 5 monclient: authenticate success, global_id 4649 2024-03-25T18:19:01.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f03383e8640 10 monclient: _renew_subs 2024-03-25T18:19:01.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f03383e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:19:01.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f03383e8640 1 -- 172.21.15.64:0/4287021535 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03301386d0 con 0x7f03301430c0 2024-03-25T18:19:01.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f03383e8640 10 client.4649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:01.039 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f03383e8640 10 monclient: _renew_subs 2024-03-25T18:19:01.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f03383e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:19:01.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f03383e8640 1 -- 172.21.15.64:0/4287021535 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03301d8060 con 0x7f03301430c0 2024-03-25T18:19:01.040 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.035+0000 7f03383e8640 1 librados: init done 2024-03-25T18:19:01.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.037+0000 7f03383e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:01.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.037+0000 7f03383e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:01.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.037+0000 7f03383e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:19:01.041 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.037+0000 7f03383e8640 1 -- 172.21.15.64:0/4287021535 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03300627b0 con 0x7f03301430c0 2024-03-25T18:19:01.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.351+0000 7f0326ffd640 1 -- 172.21.15.64:0/4287021535 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0328002070 con 0x7f03301430c0 2024-03-25T18:19:01.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.351+0000 7f0326ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:01.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.351+0000 7f0326ffd640 1 -- 172.21.15.64:0/4287021535 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f0328003b10 con 0x7f03301430c0 2024-03-25T18:19:01.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.351+0000 7f0326ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:01.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.351+0000 7f0326ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:01.352 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.351+0000 7f0326ffd640 10 monclient: dump: 2024-03-25T18:19:01.353 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:01.353 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:01.353 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:01.353 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:01.353 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:01.353 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:01.353 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:01.353 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:01.353 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:01.353 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:01.353 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.351+0000 7f0334981640 4 set_mon_vals no callback set 2024-03-25T18:19:01.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.352+0000 7f0326ffd640 1 -- 172.21.15.64:0/4287021535 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f0328003d30 con 0x7f03301430c0 2024-03-25T18:19:01.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.352+0000 7f0336184640 10 monclient: get_auth_request con 0x7f031406dc70 auth_method 0 2024-03-25T18:19:01.354 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.352+0000 7f0326ffd640 10 client.4649.objecter ms_handle_connect 0x7f031406dc70 2024-03-25T18:19:01.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.359+0000 7f0326ffd640 1 -- 172.21.15.64:0/4287021535 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f0328053850 con 0x7f03301430c0 2024-03-25T18:19:01.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.359+0000 7f0326ffd640 10 client.4649.objecter ms_dispatch 0x7f0330000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:01.360 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.359+0000 7f0326ffd640 3 client.4649.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:01.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.359+0000 7f0326ffd640 3 client.4649.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:01.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.359+0000 7f0326ffd640 20 client.4649.objecter dump_active .. 0 homeless 2024-03-25T18:19:01.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.359+0000 7f0326ffd640 1 -- 172.21.15.64:0/4287021535 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f032804f360 con 0x7f03301430c0 2024-03-25T18:19:01.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.359+0000 7f0326ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:01.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.359+0000 7f0326ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:01.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.616+0000 7f03383e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:01.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.616+0000 7f03383e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:01.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.616+0000 7f03383e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:19:01.617 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.616+0000 7f03383e8640 1 -- 172.21.15.64:0/4287021535 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f033014f240 con 0x7f03301430c0 2024-03-25T18:19:01.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.642+0000 7f0326ffd640 1 -- 172.21.15.64:0/4287021535 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f0328051290 con 0x7f03301430c0 2024-03-25T18:19:01.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.642+0000 7f0326ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:01.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.642+0000 7f0326ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:01.643 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:19:01.643 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:01.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.644+0000 7f03383e8640 10 librados: watch_flush enter 2024-03-25T18:19:01.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.644+0000 7f03383e8640 10 librados: watch_flush exit 2024-03-25T18:19:01.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.644+0000 7f03383e8640 20 client.4649.objecter shutdown clearing up homeless session... 2024-03-25T18:19:01.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.644+0000 7f03383e8640 10 client.4649.objecter successfully canceled tick 2024-03-25T18:19:01.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.644+0000 7f03383e8640 1 -- 172.21.15.64:0/4287021535 >> v1:172.21.15.64:6800/2915543638 conn(0x7f031406dc70 legacy=0x7f0314070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.644+0000 7f03383e8640 10 monclient: shutdown 2024-03-25T18:19:01.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.644+0000 7f03383e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:01.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.644+0000 7f03383e8640 1 -- 172.21.15.64:0/4287021535 >> v1:172.21.15.64:6790/0 conn(0x7f03301430c0 legacy=0x7f03301e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.644+0000 7f03383e8640 1 -- 172.21.15.64:0/4287021535 shutdown_connections 2024-03-25T18:19:01.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.644+0000 7f03383e8640 1 -- 172.21.15.64:0/4287021535 >> 172.21.15.64:0/4287021535 conn(0x7f03300b7ea0 msgr2=0x7f0330074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:01.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.644+0000 7f03383e8640 1 -- 172.21.15.64:0/4287021535 shutdown_connections 2024-03-25T18:19:01.646 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.644+0000 7f03383e8640 1 -- 172.21.15.64:0/4287021535 wait complete. 2024-03-25T18:19:01.647 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:01.644+0000 7f03383e8640 1 librados: shutdown 2024-03-25T18:19:01.658 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:03.342 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:46.454 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:03.395 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:46.508 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:03.433 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:09:46.545 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:03.447 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:09:46.525 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:03.450 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:09:46.529 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:03.486 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:46.599 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:03.487 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:09:46.565 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:03.495 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:46.608 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:03.534 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:09:46.647 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:08.465 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:08.581 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.580+0000 7f78a3850640 10 monclient: get_monmap_and_config 2024-03-25T18:19:08.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.580+0000 7f78a3850640 10 monclient: build_initial_monmap 2024-03-25T18:19:08.582 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.580+0000 7f78a3850640 10 monclient: monmap: 2024-03-25T18:19:08.582 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:19:08.582 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:08.582 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:19:08.581401+0000 2024-03-25T18:19:08.582 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:19:08.581401+0000 2024-03-25T18:19:08.582 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:08.582 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:08.582 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:19:08.583 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:19:08.583 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:19:08.583 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:08.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.581+0000 7f78a3850640 1 Processor -- start 2024-03-25T18:19:08.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.582+0000 7f78a3850640 1 -- start start 2024-03-25T18:19:08.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.582+0000 7f78a3850640 10 monclient: init 2024-03-25T18:19:08.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.583+0000 7f78a3850640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:08.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.583+0000 7f78a3850640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:08.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.583+0000 7f78a3850640 10 monclient(hunting): picked mon.noname-a con 0x7f789c0705b0 addr v1:172.21.15.64:6789/0 2024-03-25T18:19:08.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.583+0000 7f78a3850640 10 monclient(hunting): picked mon.noname-c con 0x7f789c1402d0 addr v1:172.21.15.86:6789/0 2024-03-25T18:19:08.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.583+0000 7f78a3850640 10 monclient(hunting): picked mon.noname-b con 0x7f789c149ae0 addr v1:172.21.15.64:6790/0 2024-03-25T18:19:08.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.583+0000 7f78a3850640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c0718c0 con 0x7f789c0705b0 2024-03-25T18:19:08.585 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.583+0000 7f78a3850640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c0714c0 con 0x7f789c1402d0 2024-03-25T18:19:08.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.583+0000 7f78a3850640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c072b50 con 0x7f789c149ae0 2024-03-25T18:19:08.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.583+0000 7f78a3850640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:08.586 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.583+0000 7f78a3850640 10 monclient(hunting): authenticate will time out at 7534.821688s 2024-03-25T18:19:08.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.596+0000 7f78a0deb640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f789c1402d0 0x7f789c140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:52946/0 (socket says 172.21.15.64:52946) 2024-03-25T18:19:08.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.596+0000 7f78a0deb640 1 -- 172.21.15.64:0/1572333589 learned_addr learned my addr 172.21.15.64:0/1572333589 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:19:08.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.610+0000 7f7893fff640 1 -- 172.21.15.64:0/1572333589 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136987500 0 0) 0x7f789c072b50 con 0x7f789c149ae0 2024-03-25T18:19:08.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.610+0000 7f7893fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:08.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.610+0000 7f7893fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:08.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.610+0000 7f7893fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:08.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.610+0000 7f7893fff640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:19:08.611 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.610+0000 7f7893fff640 1 -- 172.21.15.64:0/1572333589 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7888003520 con 0x7f789c149ae0 2024-03-25T18:19:08.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.612+0000 7f7893fff640 1 -- 172.21.15.64:0/1572333589 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2217179824 0 0) 0x7f789c0714c0 con 0x7f789c1402d0 2024-03-25T18:19:08.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.612+0000 7f7893fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:08.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.612+0000 7f7893fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:08.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.612+0000 7f7893fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:08.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.612+0000 7f7893fff640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:19:08.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.612+0000 7f7893fff640 1 -- 172.21.15.64:0/1572333589 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f789c072b50 con 0x7f789c1402d0 2024-03-25T18:19:08.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.618+0000 7f7893fff640 1 -- 172.21.15.64:0/1572333589 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1500954559 0 0) 0x7f7888003520 con 0x7f789c149ae0 2024-03-25T18:19:08.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.618+0000 7f7893fff640 1 -- 172.21.15.64:0/1572333589 >> v1:172.21.15.86:6789/0 conn(0x7f789c1402d0 legacy=0x7f789c140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.618+0000 7f7893fff640 1 -- 172.21.15.64:0/1572333589 >> v1:172.21.15.64:6789/0 conn(0x7f789c0705b0 legacy=0x7f789c13d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.618+0000 7f7893fff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:08.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.618+0000 7f7893fff640 1 monclient: found mon.noname-b 2024-03-25T18:19:08.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.618+0000 7f7893fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:08.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.618+0000 7f7893fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:19:08.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.618+0000 7f7893fff640 1 -- 172.21.15.64:0/1572333589 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c13dd10 con 0x7f789c149ae0 2024-03-25T18:19:08.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.618+0000 7f7893fff640 10 monclient: _finish_auth 0 2024-03-25T18:19:08.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.618+0000 7f7893fff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:08.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.618+0000 7f7893fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:08.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.618+0000 7f7893fff640 1 -- 172.21.15.64:0/1572333589 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f7894001a10 con 0x7f789c149ae0 2024-03-25T18:19:08.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.618+0000 7f78a3850640 5 monclient: authenticate success, global_id 4652 2024-03-25T18:19:08.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.618+0000 7f78a3850640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:08.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.618+0000 7f7893fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:08.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.618+0000 7f7893fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:08.621 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.618+0000 7f7893fff640 10 monclient: dump: 2024-03-25T18:19:08.621 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:08.621 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:08.621 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:08.621 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:08.621 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:08.621 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:08.621 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:08.621 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:08.622 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:08.622 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:08.622 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.618+0000 7f78a3850640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:08.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.629+0000 7f7893fff640 1 -- 172.21.15.64:0/1572333589 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7894001c30 con 0x7f789c149ae0 2024-03-25T18:19:08.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.629+0000 7f7893fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:08.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.629+0000 7f7893fff640 1 -- 172.21.15.64:0/1572333589 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f7894001fc0 con 0x7f789c149ae0 2024-03-25T18:19:08.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.629+0000 7f78a3850640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:08.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.629+0000 7f78a3850640 4 set_mon_vals no callback set 2024-03-25T18:19:08.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.629+0000 7f7893fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:08.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.629+0000 7f7893fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:08.631 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.629+0000 7f7893fff640 10 monclient: dump: 2024-03-25T18:19:08.631 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:08.631 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:08.632 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:08.632 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:08.632 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:08.632 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:08.632 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:08.632 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:08.632 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:08.632 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:08.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.630+0000 7f78a3850640 10 monclient: shutdown 2024-03-25T18:19:08.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.630+0000 7f78a3850640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:08.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.630+0000 7f78a3850640 1 -- 172.21.15.64:0/1572333589 >> v1:172.21.15.64:6790/0 conn(0x7f789c149ae0 legacy=0x7f789c14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.630+0000 7f78a3850640 1 -- 172.21.15.64:0/1572333589 shutdown_connections 2024-03-25T18:19:08.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.630+0000 7f78a3850640 1 -- 172.21.15.64:0/1572333589 >> 172.21.15.64:0/1572333589 conn(0x7f789c0b5d10 msgr2=0x7f789c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:08.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.630+0000 7f78a3850640 1 -- 172.21.15.64:0/1572333589 shutdown_connections 2024-03-25T18:19:08.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.630+0000 7f78a3850640 1 -- 172.21.15.64:0/1572333589 wait complete. 2024-03-25T18:19:08.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.631+0000 7f78a3850640 10 monclient: build_initial_monmap 2024-03-25T18:19:08.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.631+0000 7f78a3850640 10 monclient: monmap: 2024-03-25T18:19:08.635 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:19:08.636 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:08.636 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:19:08.636 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:19:08.636 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:08.636 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:19:08.636 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:19:08.636 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:19:08.636 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:19:08.636 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:08.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.631+0000 7f78a3850640 1 librados: starting msgr at 2024-03-25T18:19:08.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.631+0000 7f78a3850640 1 librados: starting objecter 2024-03-25T18:19:08.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.632+0000 7f78a3850640 1 Processor -- start 2024-03-25T18:19:08.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.632+0000 7f78a3850640 1 -- start start 2024-03-25T18:19:08.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.632+0000 7f78a3850640 1 librados: setting wanted keys 2024-03-25T18:19:08.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.632+0000 7f78a3850640 1 librados: calling monclient init 2024-03-25T18:19:08.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.632+0000 7f78a3850640 10 monclient: init 2024-03-25T18:19:08.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.632+0000 7f78a3850640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:08.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.632+0000 7f78a3850640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:08.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.632+0000 7f78a3850640 10 monclient(hunting): picked mon.noname-c con 0x7f789c0705b0 addr v1:172.21.15.64:6790/0 2024-03-25T18:19:08.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.632+0000 7f78a3850640 10 monclient(hunting): picked mon.noname-b con 0x7f789c1402d0 addr v1:172.21.15.86:6789/0 2024-03-25T18:19:08.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.632+0000 7f78a3850640 10 monclient(hunting): picked mon.noname-a con 0x7f789c149ae0 addr v1:172.21.15.64:6789/0 2024-03-25T18:19:08.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.632+0000 7f78a3850640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c141e90 con 0x7f789c149ae0 2024-03-25T18:19:08.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.633+0000 7f78a3850640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c135880 con 0x7f789c1402d0 2024-03-25T18:19:08.637 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.633+0000 7f78a3850640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c143c80 con 0x7f789c0705b0 2024-03-25T18:19:08.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.633+0000 7f78a3850640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:08.638 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.633+0000 7f78a3850640 10 monclient(hunting): authenticate will time out at 7534.871528s 2024-03-25T18:19:08.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.643+0000 7f78a1ded640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f789c149ae0 0x7f789c135170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:49394/0 (socket says 172.21.15.64:49394) 2024-03-25T18:19:08.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.643+0000 7f78a1ded640 1 -- 172.21.15.64:0/3084552380 learned_addr learned my addr 172.21.15.64:0/3084552380 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:19:08.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.644+0000 7f7891ffb640 10 client.?.objecter ms_handle_connect 0x7f789c149ae0 2024-03-25T18:19:08.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.644+0000 7f7891ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:08.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.649+0000 7f7891ffb640 10 client.?.objecter ms_handle_connect 0x7f789c1402d0 2024-03-25T18:19:08.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.649+0000 7f7891ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:08.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.650+0000 7f7891ffb640 1 -- 172.21.15.64:0/3084552380 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990433254 0 0) 0x7f789c141e90 con 0x7f789c149ae0 2024-03-25T18:19:08.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.650+0000 7f7891ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:08.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.650+0000 7f7891ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:08.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.650+0000 7f7891ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:08.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.650+0000 7f7891ffb640 10 monclient(hunting): my global_id is 4653 2024-03-25T18:19:08.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.650+0000 7f7891ffb640 1 -- 172.21.15.64:0/3084552380 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7880004eb0 con 0x7f789c149ae0 2024-03-25T18:19:08.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.654+0000 7f7891ffb640 10 client.?.objecter ms_handle_connect 0x7f789c0705b0 2024-03-25T18:19:08.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.654+0000 7f7891ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:08.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.661+0000 7f7891ffb640 1 -- 172.21.15.64:0/3084552380 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 791423868 0 0) 0x7f7880004eb0 con 0x7f789c149ae0 2024-03-25T18:19:08.663 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.661+0000 7f7891ffb640 1 -- 172.21.15.64:0/3084552380 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f789c141e90 con 0x7f789c149ae0 2024-03-25T18:19:08.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.671+0000 7f7891ffb640 1 -- 172.21.15.64:0/3084552380 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f7894001f40 con 0x7f789c149ae0 2024-03-25T18:19:08.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.671+0000 7f7891ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:08.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.671+0000 7f7891ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:08.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.671+0000 7f7891ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:08.673 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:08.673 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:08.673 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:08.673 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:08.673 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:08.673 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:08.673 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:08.674 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:08.674 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:08.674 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:08.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.671+0000 7f7891ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:08.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.672+0000 7f7891ffb640 1 -- 172.21.15.64:0/3084552380 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010890292 0 0) 0x7f789c135880 con 0x7f789c1402d0 2024-03-25T18:19:08.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.672+0000 7f7891ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:08.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.672+0000 7f7891ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:08.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.672+0000 7f7891ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:08.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.672+0000 7f7891ffb640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:19:08.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.672+0000 7f7891ffb640 1 -- 172.21.15.64:0/3084552380 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7880004eb0 con 0x7f789c1402d0 2024-03-25T18:19:08.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.680+0000 7f7891ffb640 1 -- 172.21.15.64:0/3084552380 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 907239094 0 0) 0x7f789c141e90 con 0x7f789c149ae0 2024-03-25T18:19:08.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.680+0000 7f7891ffb640 1 -- 172.21.15.64:0/3084552380 >> v1:172.21.15.64:6790/0 conn(0x7f789c0705b0 legacy=0x7f789c143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.682 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.680+0000 7f7891ffb640 1 -- 172.21.15.64:0/3084552380 >> v1:172.21.15.86:6789/0 conn(0x7f789c1402d0 legacy=0x7f789c134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.680+0000 7f7891ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:08.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.680+0000 7f7891ffb640 1 monclient: found mon.a 2024-03-25T18:19:08.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.680+0000 7f7891ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:08.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.680+0000 7f7891ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:19:08.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.680+0000 7f7891ffb640 1 -- 172.21.15.64:0/3084552380 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c1420e0 con 0x7f789c149ae0 2024-03-25T18:19:08.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.680+0000 7f7891ffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:08.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.680+0000 7f7891ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:08.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.680+0000 7f7891ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:08.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.680+0000 7f78a3850640 5 monclient: authenticate success, global_id 4653 2024-03-25T18:19:08.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.680+0000 7f78a3850640 10 monclient: _renew_subs 2024-03-25T18:19:08.683 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.680+0000 7f78a3850640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:19:08.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.680+0000 7f78a3850640 1 -- 172.21.15.64:0/3084552380 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789c1423c0 con 0x7f789c149ae0 2024-03-25T18:19:08.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.681+0000 7f78a3850640 10 client.4653.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:08.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.681+0000 7f78a3850640 10 monclient: _renew_subs 2024-03-25T18:19:08.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.681+0000 7f78a3850640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:19:08.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.681+0000 7f78a3850640 1 -- 172.21.15.64:0/3084552380 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f789c142b30 con 0x7f789c149ae0 2024-03-25T18:19:08.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.681+0000 7f78a3850640 1 librados: init done 2024-03-25T18:19:08.684 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.682+0000 7f78a3850640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:08.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.682+0000 7f78a3850640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:08.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.682+0000 7f78a3850640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:19:08.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:08.682+0000 7f78a3850640 1 -- 172.21.15.64:0/3084552380 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f789c0627b0 con 0x7f789c149ae0 2024-03-25T18:19:09.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.002+0000 7f7891ffb640 1 -- 172.21.15.64:0/3084552380 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78940020a0 con 0x7f789c149ae0 2024-03-25T18:19:09.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.002+0000 7f7891ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:09.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.002+0000 7f7891ffb640 1 -- 172.21.15.64:0/3084552380 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f7894003e40 con 0x7f789c149ae0 2024-03-25T18:19:09.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.002+0000 7f7891ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:09.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.002+0000 7f7891ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:09.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.002+0000 7f7891ffb640 10 monclient: dump: 2024-03-25T18:19:09.004 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:09.004 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:09.004 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:09.004 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:09.004 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:09.004 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:09.004 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:09.004 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:09.004 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:09.004 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:09.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.002+0000 7f78937fe640 4 set_mon_vals no callback set 2024-03-25T18:19:09.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.003+0000 7f7891ffb640 1 -- 172.21.15.64:0/3084552380 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f7894004060 con 0x7f789c149ae0 2024-03-25T18:19:09.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.003+0000 7f78a15ec640 10 monclient: get_auth_request con 0x7f788006dc90 auth_method 0 2024-03-25T18:19:09.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.003+0000 7f7891ffb640 10 client.4653.objecter ms_handle_connect 0x7f788006dc90 2024-03-25T18:19:09.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.010+0000 7f7891ffb640 1 -- 172.21.15.64:0/3084552380 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7f7894054c40 con 0x7f789c149ae0 2024-03-25T18:19:09.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.010+0000 7f7891ffb640 10 client.4653.objecter ms_dispatch 0x7f789c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:09.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.010+0000 7f7891ffb640 3 client.4653.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:09.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.010+0000 7f7891ffb640 3 client.4653.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:09.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.010+0000 7f7891ffb640 20 client.4653.objecter dump_active .. 0 homeless 2024-03-25T18:19:09.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.010+0000 7f7891ffb640 1 -- 172.21.15.64:0/3084552380 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7894050750 con 0x7f789c149ae0 2024-03-25T18:19:09.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.010+0000 7f7891ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:09.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.010+0000 7f7891ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:09.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.240+0000 7f78a3850640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:09.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.240+0000 7f78a3850640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:09.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.240+0000 7f78a3850640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:19:09.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.240+0000 7f78a3850640 1 -- 172.21.15.64:0/3084552380 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f789c140a10 con 0x7f789c149ae0 2024-03-25T18:19:09.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.269+0000 7f7891ffb640 1 -- 172.21.15.64:0/3084552380 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7f7894052680 con 0x7f789c149ae0 2024-03-25T18:19:09.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.269+0000 7f7891ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:09.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.269+0000 7f7891ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:09.270 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:19:09.270 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:09.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.271+0000 7f78a3850640 10 librados: watch_flush enter 2024-03-25T18:19:09.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.271+0000 7f78a3850640 10 librados: watch_flush exit 2024-03-25T18:19:09.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.271+0000 7f78a3850640 20 client.4653.objecter shutdown clearing up homeless session... 2024-03-25T18:19:09.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.271+0000 7f78a3850640 10 client.4653.objecter successfully canceled tick 2024-03-25T18:19:09.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.271+0000 7f78a3850640 1 -- 172.21.15.64:0/3084552380 >> v1:172.21.15.64:6800/2915543638 conn(0x7f788006dc90 legacy=0x7f7880070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:09.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.272+0000 7f78a3850640 10 monclient: shutdown 2024-03-25T18:19:09.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.272+0000 7f78a3850640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:09.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.272+0000 7f78a3850640 1 -- 172.21.15.64:0/3084552380 >> v1:172.21.15.64:6789/0 conn(0x7f789c149ae0 legacy=0x7f789c135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:09.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.272+0000 7f78a3850640 1 -- 172.21.15.64:0/3084552380 shutdown_connections 2024-03-25T18:19:09.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.272+0000 7f78a3850640 1 -- 172.21.15.64:0/3084552380 >> 172.21.15.64:0/3084552380 conn(0x7f789c0b5d10 msgr2=0x7f789c13b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:09.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.272+0000 7f78a3850640 1 -- 172.21.15.64:0/3084552380 shutdown_connections 2024-03-25T18:19:09.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.272+0000 7f78a3850640 1 -- 172.21.15.64:0/3084552380 wait complete. 2024-03-25T18:19:09.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:09.272+0000 7f78a3850640 1 librados: shutdown 2024-03-25T18:19:09.285 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:16.091 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:16.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.205+0000 7fb1b7f93640 10 monclient: get_monmap_and_config 2024-03-25T18:19:16.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.205+0000 7fb1b7f93640 10 monclient: build_initial_monmap 2024-03-25T18:19:16.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.205+0000 7fb1b7f93640 10 monclient: monmap: 2024-03-25T18:19:16.206 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:19:16.206 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:16.206 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:19:16.206351+0000 2024-03-25T18:19:16.206 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:19:16.206351+0000 2024-03-25T18:19:16.206 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:16.207 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:16.207 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:19:16.207 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:19:16.207 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:19:16.207 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:16.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.206+0000 7fb1b7f93640 1 Processor -- start 2024-03-25T18:19:16.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.206+0000 7fb1b7f93640 1 -- start start 2024-03-25T18:19:16.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.206+0000 7fb1b7f93640 10 monclient: init 2024-03-25T18:19:16.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.206+0000 7fb1b7f93640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:16.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.206+0000 7fb1b7f93640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:16.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.206+0000 7fb1b7f93640 10 monclient(hunting): picked mon.noname-b con 0x7fb1b00b4d60 addr v1:172.21.15.64:6790/0 2024-03-25T18:19:16.208 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.206+0000 7fb1b7f93640 10 monclient(hunting): picked mon.noname-a con 0x7fb1b00b1730 addr v1:172.21.15.64:6789/0 2024-03-25T18:19:16.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.207+0000 7fb1b7f93640 10 monclient(hunting): picked mon.noname-c con 0x7fb1b014be90 addr v1:172.21.15.86:6789/0 2024-03-25T18:19:16.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.207+0000 7fb1b7f93640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b00af940 con 0x7fb1b00b1730 2024-03-25T18:19:16.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.207+0000 7fb1b7f93640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b00b0a70 con 0x7fb1b014be90 2024-03-25T18:19:16.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.207+0000 7fb1b7f93640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b00b0d50 con 0x7fb1b00b4d60 2024-03-25T18:19:16.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.207+0000 7fb1b7f93640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:16.209 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.207+0000 7fb1b7f93640 10 monclient(hunting): authenticate will time out at 7542.445595s 2024-03-25T18:19:16.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.213+0000 7fb1b552e640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fb1b00b1730 0x7fb1b00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:54036/0 (socket says 172.21.15.64:54036) 2024-03-25T18:19:16.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.213+0000 7fb1b552e640 1 -- 172.21.15.64:0/950795446 learned_addr learned my addr 172.21.15.64:0/950795446 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:19:16.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.235+0000 7fb1b4d2d640 1 -- 172.21.15.64:0/950795446 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823970959 0 0) 0x7fb1b00af940 con 0x7fb1b00b1730 2024-03-25T18:19:16.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.235+0000 7fb1b4d2d640 10 monclient(hunting): _negotiate 2024-03-25T18:19:16.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.235+0000 7fb1b4d2d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:16.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.235+0000 7fb1b4d2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:16.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.235+0000 7fb1b4d2d640 10 monclient(hunting): my global_id is 4656 2024-03-25T18:19:16.237 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.235+0000 7fb1b4d2d640 1 -- 172.21.15.64:0/950795446 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb19c003520 con 0x7fb1b00b1730 2024-03-25T18:19:16.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.237+0000 7fb1b4d2d640 1 -- 172.21.15.64:0/950795446 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837161622 0 0) 0x7fb1b00b0a70 con 0x7fb1b014be90 2024-03-25T18:19:16.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.237+0000 7fb1b4d2d640 10 monclient(hunting): _negotiate 2024-03-25T18:19:16.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.237+0000 7fb1b4d2d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:16.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.237+0000 7fb1b4d2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:16.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.237+0000 7fb1b4d2d640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:19:16.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.237+0000 7fb1b4d2d640 1 -- 172.21.15.64:0/950795446 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1b00af940 con 0x7fb1b014be90 2024-03-25T18:19:16.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.245+0000 7fb1b4d2d640 1 -- 172.21.15.64:0/950795446 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3433361635 0 0) 0x7fb19c003520 con 0x7fb1b00b1730 2024-03-25T18:19:16.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.245+0000 7fb1b4d2d640 1 -- 172.21.15.64:0/950795446 >> v1:172.21.15.64:6790/0 conn(0x7fb1b00b4d60 legacy=0x7fb1b013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:19:16.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.245+0000 7fb1b4d2d640 1 -- 172.21.15.64:0/950795446 >> v1:172.21.15.86:6789/0 conn(0x7fb1b014be90 legacy=0x7fb1b014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.245+0000 7fb1b4d2d640 10 monclient: _finish_hunting 0 2024-03-25T18:19:16.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.245+0000 7fb1b4d2d640 1 monclient: found mon.noname-a 2024-03-25T18:19:16.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.245+0000 7fb1b4d2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:16.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.245+0000 7fb1b4d2d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:19:16.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.245+0000 7fb1b4d2d640 1 -- 172.21.15.64:0/950795446 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b00b1030 con 0x7fb1b00b1730 2024-03-25T18:19:16.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.245+0000 7fb1b4d2d640 10 monclient: _finish_auth 0 2024-03-25T18:19:16.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.245+0000 7fb1b4d2d640 10 monclient: _check_auth_tickets 2024-03-25T18:19:16.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.245+0000 7fb1b4d2d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:16.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.245+0000 7fb1b4d2d640 1 -- 172.21.15.64:0/950795446 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb1a0001690 con 0x7fb1b00b1730 2024-03-25T18:19:16.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.245+0000 7fb1b4d2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:16.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.245+0000 7fb1b4d2d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:16.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.245+0000 7fb1b4d2d640 10 monclient: dump: 2024-03-25T18:19:16.247 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:16.247 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:16.247 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:16.247 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:16.247 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:16.247 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:16.247 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:16.247 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:16.247 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:16.247 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:16.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.245+0000 7fb1b7f93640 5 monclient: authenticate success, global_id 4656 2024-03-25T18:19:16.247 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.245+0000 7fb1b7f93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:16.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.254+0000 7fb1b4d2d640 1 -- 172.21.15.64:0/950795446 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1a00017f0 con 0x7fb1b00b1730 2024-03-25T18:19:16.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.254+0000 7fb1b4d2d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:16.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.254+0000 7fb1b4d2d640 1 -- 172.21.15.64:0/950795446 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb1a0001c00 con 0x7fb1b00b1730 2024-03-25T18:19:16.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.254+0000 7fb1b7f93640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:16.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.254+0000 7fb1b7f93640 4 set_mon_vals no callback set 2024-03-25T18:19:16.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.254+0000 7fb1b4d2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:16.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.254+0000 7fb1b4d2d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:16.255 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.254+0000 7fb1b4d2d640 10 monclient: dump: 2024-03-25T18:19:16.255 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:16.255 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:16.255 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:16.255 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:16.255 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:16.255 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:16.256 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:16.256 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:16.256 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:16.256 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:16.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.254+0000 7fb1b7f93640 10 monclient: shutdown 2024-03-25T18:19:16.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.254+0000 7fb1b7f93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:16.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.254+0000 7fb1b7f93640 1 -- 172.21.15.64:0/950795446 >> v1:172.21.15.64:6789/0 conn(0x7fb1b00b1730 legacy=0x7fb1b00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.254+0000 7fb1b7f93640 1 -- 172.21.15.64:0/950795446 shutdown_connections 2024-03-25T18:19:16.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.254+0000 7fb1b7f93640 1 -- 172.21.15.64:0/950795446 >> 172.21.15.64:0/950795446 conn(0x7fb1b00617a0 msgr2=0x7fb1b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:16.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.255+0000 7fb1b7f93640 1 -- 172.21.15.64:0/950795446 shutdown_connections 2024-03-25T18:19:16.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.255+0000 7fb1b7f93640 1 -- 172.21.15.64:0/950795446 wait complete. 2024-03-25T18:19:16.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.255+0000 7fb1b7f93640 10 monclient: build_initial_monmap 2024-03-25T18:19:16.258 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.255+0000 7fb1b7f93640 10 monclient: monmap: 2024-03-25T18:19:16.259 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:19:16.259 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:16.259 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:19:16.259 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:19:16.259 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:16.259 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:19:16.259 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:19:16.259 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:19:16.259 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:19:16.259 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:16.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.256+0000 7fb1b7f93640 1 librados: starting msgr at 2024-03-25T18:19:16.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.256+0000 7fb1b7f93640 1 librados: starting objecter 2024-03-25T18:19:16.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.256+0000 7fb1b7f93640 1 Processor -- start 2024-03-25T18:19:16.259 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.256+0000 7fb1b7f93640 1 -- start start 2024-03-25T18:19:16.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.256+0000 7fb1b7f93640 1 librados: setting wanted keys 2024-03-25T18:19:16.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.256+0000 7fb1b7f93640 1 librados: calling monclient init 2024-03-25T18:19:16.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.256+0000 7fb1b7f93640 10 monclient: init 2024-03-25T18:19:16.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.256+0000 7fb1b7f93640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:16.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.256+0000 7fb1b7f93640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:16.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.256+0000 7fb1b7f93640 10 monclient(hunting): picked mon.noname-a con 0x7fb1b00b1730 addr v1:172.21.15.64:6789/0 2024-03-25T18:19:16.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.257+0000 7fb1b7f93640 10 monclient(hunting): picked mon.noname-b con 0x7fb1b00b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:19:16.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.257+0000 7fb1b7f93640 10 monclient(hunting): picked mon.noname-c con 0x7fb1b014be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:19:16.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.257+0000 7fb1b7f93640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b01409c0 con 0x7fb1b00b1730 2024-03-25T18:19:16.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.257+0000 7fb1b7f93640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b0132620 con 0x7fb1b00b4d60 2024-03-25T18:19:16.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.257+0000 7fb1b7f93640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b01327b0 con 0x7fb1b014be90 2024-03-25T18:19:16.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.257+0000 7fb1b7f93640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:16.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.257+0000 7fb1b7f93640 10 monclient(hunting): authenticate will time out at 7542.495504s 2024-03-25T18:19:16.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.265+0000 7fb1b5d2f640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fb1b00b1730 0x7fb1b01383c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:54056/0 (socket says 172.21.15.64:54056) 2024-03-25T18:19:16.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.265+0000 7fb1b5d2f640 1 -- 172.21.15.64:0/1729131663 learned_addr learned my addr 172.21.15.64:0/1729131663 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:19:16.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.268+0000 7fb1a67fc640 10 client.?.objecter ms_handle_connect 0x7fb1b00b1730 2024-03-25T18:19:16.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.268+0000 7fb1a67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:16.271 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.270+0000 7fb1a67fc640 10 client.?.objecter ms_handle_connect 0x7fb1b00b4d60 2024-03-25T18:19:16.271 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.270+0000 7fb1a67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:16.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.272+0000 7fb1a67fc640 10 client.?.objecter ms_handle_connect 0x7fb1b014be90 2024-03-25T18:19:16.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.272+0000 7fb1a67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:16.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.273+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860700917 0 0) 0x7fb1b01409c0 con 0x7fb1b00b1730 2024-03-25T18:19:16.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.273+0000 7fb1a67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:16.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.273+0000 7fb1a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:16.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.273+0000 7fb1a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:16.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.273+0000 7fb1a67fc640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:19:16.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.273+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb194004eb0 con 0x7fb1b00b1730 2024-03-25T18:19:16.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.276+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778795636 0 0) 0x7fb1b01327b0 con 0x7fb1b014be90 2024-03-25T18:19:16.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.276+0000 7fb1a67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:16.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.276+0000 7fb1a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:16.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.276+0000 7fb1a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:16.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.276+0000 7fb1a67fc640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:19:16.277 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.276+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1b01409c0 con 0x7fb1b014be90 2024-03-25T18:19:16.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.289+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546740357 0 0) 0x7fb1b0132620 con 0x7fb1b00b4d60 2024-03-25T18:19:16.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.289+0000 7fb1a67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:16.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.289+0000 7fb1a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:16.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.289+0000 7fb1a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:16.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.289+0000 7fb1a67fc640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:19:16.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.289+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1b01327b0 con 0x7fb1b00b4d60 2024-03-25T18:19:16.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.296+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 926207879 0 0) 0x7fb1b01409c0 con 0x7fb1b014be90 2024-03-25T18:19:16.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.296+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1b0132620 con 0x7fb1b014be90 2024-03-25T18:19:16.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.296+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb1a8001b50 con 0x7fb1b014be90 2024-03-25T18:19:16.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.296+0000 7fb1a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:16.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.296+0000 7fb1a67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:16.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.296+0000 7fb1a67fc640 10 monclient(hunting): dump: 2024-03-25T18:19:16.298 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:16.298 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:16.298 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:16.298 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:16.298 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:16.298 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:16.298 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:16.298 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:16.298 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:16.298 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:16.298 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.296+0000 7fb1a67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:16.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.300+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 707542474 0 0) 0x7fb194004eb0 con 0x7fb1b00b1730 2024-03-25T18:19:16.301 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.300+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1b01409c0 con 0x7fb1b00b1730 2024-03-25T18:19:16.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.300+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb1ac001960 con 0x7fb1b00b1730 2024-03-25T18:19:16.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.300+0000 7fb1a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:16.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.300+0000 7fb1a67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:16.302 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.300+0000 7fb1a67fc640 10 monclient(hunting): dump: 2024-03-25T18:19:16.302 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:16.302 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:16.302 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:16.302 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:16.302 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:16.302 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:16.302 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:16.302 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:16.302 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:16.302 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:16.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.303+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1385078354 0 0) 0x7fb1b01327b0 con 0x7fb1b00b4d60 2024-03-25T18:19:16.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.303+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb194004eb0 con 0x7fb1b00b4d60 2024-03-25T18:19:16.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.303+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb1a0001b00 con 0x7fb1b00b4d60 2024-03-25T18:19:16.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.303+0000 7fb1a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:16.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.303+0000 7fb1a67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:16.304 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.303+0000 7fb1a67fc640 10 monclient(hunting): dump: 2024-03-25T18:19:16.304 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:16.304 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:16.304 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:16.304 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:16.304 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:16.304 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:16.304 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:16.304 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:16.304 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:16.305 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:16.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2154461330 0 0) 0x7fb1b0132620 con 0x7fb1b014be90 2024-03-25T18:19:16.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 >> v1:172.21.15.86:6789/0 conn(0x7fb1b00b4d60 legacy=0x7fb1b0131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 >> v1:172.21.15.64:6789/0 conn(0x7fb1b00b1730 legacy=0x7fb1b01383c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1a67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:16.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1a67fc640 1 monclient: found mon.c 2024-03-25T18:19:16.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:16.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1a67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:19:16.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b0137f70 con 0x7fb1b014be90 2024-03-25T18:19:16.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1a67fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:16.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1a67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:16.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:16.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1b7f93640 5 monclient: authenticate success, global_id 4658 2024-03-25T18:19:16.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1b7f93640 10 monclient: _renew_subs 2024-03-25T18:19:16.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1b7f93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:19:16.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1b7f93640 1 -- 172.21.15.64:0/1729131663 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1b01381e0 con 0x7fb1b014be90 2024-03-25T18:19:16.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1b7f93640 10 client.4658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:16.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1b7f93640 10 monclient: _renew_subs 2024-03-25T18:19:16.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1b7f93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:19:16.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1b7f93640 1 -- 172.21.15.64:0/1729131663 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1b0145340 con 0x7fb1b014be90 2024-03-25T18:19:16.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.306+0000 7fb1b7f93640 1 librados: init done 2024-03-25T18:19:16.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.308+0000 7fb1b7f93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:16.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.308+0000 7fb1b7f93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:16.309 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.308+0000 7fb1b7f93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:19:16.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.308+0000 7fb1b7f93640 1 -- 172.21.15.64:0/1729131663 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1b0140e30 con 0x7fb1b014be90 2024-03-25T18:19:16.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.639+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1a8002000 con 0x7fb1b014be90 2024-03-25T18:19:16.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.639+0000 7fb1a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:16.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.639+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb1a8003970 con 0x7fb1b014be90 2024-03-25T18:19:16.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.639+0000 7fb1a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:16.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.640+0000 7fb1a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:16.641 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.640+0000 7fb1a67fc640 10 monclient: dump: 2024-03-25T18:19:16.641 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:16.641 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:16.641 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:16.641 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:16.641 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:16.641 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:16.641 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:16.641 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:16.642 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:16.642 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:16.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.640+0000 7fb1a7fff640 4 set_mon_vals no callback set 2024-03-25T18:19:16.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.641+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fb1a8019940 con 0x7fb1b014be90 2024-03-25T18:19:16.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.641+0000 7fb1b5d2f640 10 monclient: get_auth_request con 0x7fb19406e0f0 auth_method 0 2024-03-25T18:19:16.642 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.641+0000 7fb1a67fc640 10 client.4658.objecter ms_handle_connect 0x7fb19406e0f0 2024-03-25T18:19:16.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.647+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fb1a8053530 con 0x7fb1b014be90 2024-03-25T18:19:16.648 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.647+0000 7fb1a67fc640 10 client.4658.objecter ms_dispatch 0x7fb1b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:16.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.647+0000 7fb1a67fc640 3 client.4658.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:16.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.647+0000 7fb1a67fc640 3 client.4658.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:16.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.647+0000 7fb1a67fc640 20 client.4658.objecter dump_active .. 0 homeless 2024-03-25T18:19:16.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.647+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1a804f040 con 0x7fb1b014be90 2024-03-25T18:19:16.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.647+0000 7fb1a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:16.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.647+0000 7fb1a67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:16.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.897+0000 7fb1b7f93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:16.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.897+0000 7fb1b7f93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:16.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.897+0000 7fb1b7f93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:19:16.898 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.897+0000 7fb1b7f93640 1 -- 172.21.15.64:0/1729131663 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1b0140b50 con 0x7fb1b014be90 2024-03-25T18:19:16.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.937+0000 7fb1a67fc640 1 -- 172.21.15.64:0/1729131663 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fb1a8050f70 con 0x7fb1b014be90 2024-03-25T18:19:16.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.937+0000 7fb1a67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:16.938 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.937+0000 7fb1a67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:16.938 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:19:16.938 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:16.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.939+0000 7fb1b7f93640 10 librados: watch_flush enter 2024-03-25T18:19:16.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.939+0000 7fb1b7f93640 10 librados: watch_flush exit 2024-03-25T18:19:16.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.939+0000 7fb1b7f93640 20 client.4658.objecter shutdown clearing up homeless session... 2024-03-25T18:19:16.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.939+0000 7fb1b7f93640 10 client.4658.objecter successfully canceled tick 2024-03-25T18:19:16.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.939+0000 7fb1b7f93640 1 -- 172.21.15.64:0/1729131663 >> v1:172.21.15.64:6800/2915543638 conn(0x7fb19406e0f0 legacy=0x7fb1940705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.939+0000 7fb1b7f93640 10 monclient: shutdown 2024-03-25T18:19:16.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.939+0000 7fb1b7f93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:16.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.939+0000 7fb1b7f93640 1 -- 172.21.15.64:0/1729131663 >> v1:172.21.15.64:6790/0 conn(0x7fb1b014be90 legacy=0x7fb1b01423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.939+0000 7fb1b7f93640 1 -- 172.21.15.64:0/1729131663 shutdown_connections 2024-03-25T18:19:16.941 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.939+0000 7fb1b7f93640 1 -- 172.21.15.64:0/1729131663 >> 172.21.15.64:0/1729131663 conn(0x7fb1b00617a0 msgr2=0x7fb1b013c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:16.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.939+0000 7fb1b7f93640 1 -- 172.21.15.64:0/1729131663 shutdown_connections 2024-03-25T18:19:16.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.939+0000 7fb1b7f93640 1 -- 172.21.15.64:0/1729131663 wait complete. 2024-03-25T18:19:16.942 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:16.940+0000 7fb1b7f93640 1 librados: shutdown 2024-03-25T18:19:16.954 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:23.445 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:10:06.557 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:23.453 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:10:06.566 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:23.491 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:10:06.603 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:23.494 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:10:06.572 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:23.505 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:10:06.584 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:23.516 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:10:06.629 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:23.524 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:10:06.636 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:23.534 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:10:06.613 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:23.556 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:10:06.668 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:23.759 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:23.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.874+0000 7fa4d6348640 10 monclient: get_monmap_and_config 2024-03-25T18:19:23.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.874+0000 7fa4d6348640 10 monclient: build_initial_monmap 2024-03-25T18:19:23.876 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.874+0000 7fa4d6348640 10 monclient: monmap: 2024-03-25T18:19:23.876 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:19:23.877 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:23.877 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:19:23.876465+0000 2024-03-25T18:19:23.877 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:19:23.876465+0000 2024-03-25T18:19:23.877 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:23.877 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:23.877 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:19:23.877 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:19:23.877 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:19:23.877 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:23.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.875+0000 7fa4d6348640 1 Processor -- start 2024-03-25T18:19:23.878 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.875+0000 7fa4d6348640 1 -- start start 2024-03-25T18:19:23.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.875+0000 7fa4d6348640 10 monclient: init 2024-03-25T18:19:23.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.876+0000 7fa4d6348640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:23.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.876+0000 7fa4d6348640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:23.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.876+0000 7fa4d6348640 10 monclient(hunting): picked mon.noname-a con 0x7fa4d00b4d60 addr v1:172.21.15.64:6789/0 2024-03-25T18:19:23.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.876+0000 7fa4d6348640 10 monclient(hunting): picked mon.noname-c con 0x7fa4d00b1730 addr v1:172.21.15.86:6789/0 2024-03-25T18:19:23.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.876+0000 7fa4d6348640 10 monclient(hunting): picked mon.noname-b con 0x7fa4d014be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:19:23.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.876+0000 7fa4d6348640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d00af940 con 0x7fa4d00b4d60 2024-03-25T18:19:23.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.876+0000 7fa4d6348640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d00b0a70 con 0x7fa4d00b1730 2024-03-25T18:19:23.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.876+0000 7fa4d6348640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d00b0d50 con 0x7fa4d014be90 2024-03-25T18:19:23.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.876+0000 7fa4d6348640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:23.879 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.876+0000 7fa4d6348640 10 monclient(hunting): authenticate will time out at 7550.116158s 2024-03-25T18:19:23.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.895+0000 7fa4cffff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fa4d00b4d60 0x7fa4d013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:39126/0 (socket says 172.21.15.64:39126) 2024-03-25T18:19:23.897 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.895+0000 7fa4cffff640 1 -- 172.21.15.64:0/2146982109 learned_addr learned my addr 172.21.15.64:0/2146982109 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:19:23.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.907+0000 7fa4ceffd640 1 -- 172.21.15.64:0/2146982109 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745381848 0 0) 0x7fa4d00af940 con 0x7fa4d00b4d60 2024-03-25T18:19:23.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.907+0000 7fa4ceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:23.909 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.907+0000 7fa4ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:23.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.907+0000 7fa4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:23.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.907+0000 7fa4ceffd640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:19:23.910 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.907+0000 7fa4ceffd640 1 -- 172.21.15.64:0/2146982109 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4b8003520 con 0x7fa4d00b4d60 2024-03-25T18:19:23.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.913+0000 7fa4ceffd640 1 -- 172.21.15.64:0/2146982109 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889896130 0 0) 0x7fa4d00b0d50 con 0x7fa4d014be90 2024-03-25T18:19:23.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.913+0000 7fa4ceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:23.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.913+0000 7fa4ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:23.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.913+0000 7fa4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:23.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.913+0000 7fa4ceffd640 10 monclient(hunting): my global_id is 4661 2024-03-25T18:19:23.915 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.913+0000 7fa4ceffd640 1 -- 172.21.15.64:0/2146982109 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d00af940 con 0x7fa4d014be90 2024-03-25T18:19:23.918 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.916+0000 7fa4ceffd640 1 -- 172.21.15.64:0/2146982109 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3039987780 0 0) 0x7fa4d00b0a70 con 0x7fa4d00b1730 2024-03-25T18:19:23.918 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.916+0000 7fa4ceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:23.918 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.916+0000 7fa4ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:23.918 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.916+0000 7fa4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:23.918 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.916+0000 7fa4ceffd640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:19:23.919 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.916+0000 7fa4ceffd640 1 -- 172.21.15.64:0/2146982109 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d00b0d50 con 0x7fa4d00b1730 2024-03-25T18:19:23.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.926+0000 7fa4ceffd640 1 -- 172.21.15.64:0/2146982109 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1707342857 0 0) 0x7fa4d00af940 con 0x7fa4d014be90 2024-03-25T18:19:23.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.926+0000 7fa4ceffd640 1 -- 172.21.15.64:0/2146982109 >> v1:172.21.15.86:6789/0 conn(0x7fa4d00b1730 legacy=0x7fa4d00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.926+0000 7fa4ceffd640 1 -- 172.21.15.64:0/2146982109 >> v1:172.21.15.64:6789/0 conn(0x7fa4d00b4d60 legacy=0x7fa4d013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.926+0000 7fa4ceffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:23.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.926+0000 7fa4ceffd640 1 monclient: found mon.noname-b 2024-03-25T18:19:23.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.926+0000 7fa4ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:23.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.926+0000 7fa4ceffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:19:23.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.926+0000 7fa4ceffd640 1 -- 172.21.15.64:0/2146982109 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d00b1030 con 0x7fa4d014be90 2024-03-25T18:19:23.929 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.926+0000 7fa4ceffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:23.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.926+0000 7fa4ceffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:23.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.926+0000 7fa4ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:23.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.926+0000 7fa4ceffd640 1 -- 172.21.15.64:0/2146982109 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fa4c80019f0 con 0x7fa4d014be90 2024-03-25T18:19:23.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.926+0000 7fa4ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:23.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.926+0000 7fa4ceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:23.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.926+0000 7fa4ceffd640 10 monclient: dump: 2024-03-25T18:19:23.930 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:23.930 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:23.930 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:23.930 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:23.930 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:23.930 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:23.931 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:23.931 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:23.931 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:23.931 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:23.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.926+0000 7fa4d6348640 5 monclient: authenticate success, global_id 4661 2024-03-25T18:19:23.931 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.926+0000 7fa4d6348640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:23.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.942+0000 7fa4ceffd640 1 -- 172.21.15.64:0/2146982109 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4c8001b50 con 0x7fa4d014be90 2024-03-25T18:19:23.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.942+0000 7fa4ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:23.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.942+0000 7fa4ceffd640 1 -- 172.21.15.64:0/2146982109 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fa4c8001f60 con 0x7fa4d014be90 2024-03-25T18:19:23.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.942+0000 7fa4d6348640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:23.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.942+0000 7fa4d6348640 4 set_mon_vals no callback set 2024-03-25T18:19:23.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.942+0000 7fa4ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:23.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.942+0000 7fa4ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:23.944 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.942+0000 7fa4ceffd640 10 monclient: dump: 2024-03-25T18:19:23.944 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:23.944 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:23.944 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:23.944 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:23.945 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:23.945 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:23.945 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:23.945 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:23.945 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:23.945 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:23.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.942+0000 7fa4d6348640 10 monclient: shutdown 2024-03-25T18:19:23.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.942+0000 7fa4d6348640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:23.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.942+0000 7fa4d6348640 1 -- 172.21.15.64:0/2146982109 >> v1:172.21.15.64:6790/0 conn(0x7fa4d014be90 legacy=0x7fa4d014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.943+0000 7fa4d6348640 1 -- 172.21.15.64:0/2146982109 shutdown_connections 2024-03-25T18:19:23.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.943+0000 7fa4d6348640 1 -- 172.21.15.64:0/2146982109 >> 172.21.15.64:0/2146982109 conn(0x7fa4d00617a0 msgr2=0x7fa4d0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:23.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.943+0000 7fa4d6348640 1 -- 172.21.15.64:0/2146982109 shutdown_connections 2024-03-25T18:19:23.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.943+0000 7fa4d6348640 1 -- 172.21.15.64:0/2146982109 wait complete. 2024-03-25T18:19:23.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.944+0000 7fa4d6348640 10 monclient: build_initial_monmap 2024-03-25T18:19:23.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.944+0000 7fa4d6348640 10 monclient: monmap: 2024-03-25T18:19:23.947 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:19:23.947 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:23.947 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:19:23.947 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:19:23.947 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:23.947 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:19:23.947 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:19:23.947 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:19:23.947 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:19:23.947 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:23.947 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.944+0000 7fa4d6348640 1 librados: starting msgr at 2024-03-25T18:19:23.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.944+0000 7fa4d6348640 1 librados: starting objecter 2024-03-25T18:19:23.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.944+0000 7fa4d6348640 1 Processor -- start 2024-03-25T18:19:23.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.944+0000 7fa4d6348640 1 -- start start 2024-03-25T18:19:23.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.944+0000 7fa4d6348640 1 librados: setting wanted keys 2024-03-25T18:19:23.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.944+0000 7fa4d6348640 1 librados: calling monclient init 2024-03-25T18:19:23.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.944+0000 7fa4d6348640 10 monclient: init 2024-03-25T18:19:23.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.945+0000 7fa4d6348640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:23.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.945+0000 7fa4d6348640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:23.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.945+0000 7fa4d6348640 10 monclient(hunting): picked mon.noname-c con 0x7fa4d00b1730 addr v1:172.21.15.64:6790/0 2024-03-25T18:19:23.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.945+0000 7fa4d6348640 10 monclient(hunting): picked mon.noname-b con 0x7fa4d00b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:19:23.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.945+0000 7fa4d6348640 10 monclient(hunting): picked mon.noname-a con 0x7fa4d014be90 addr v1:172.21.15.64:6789/0 2024-03-25T18:19:23.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.945+0000 7fa4d6348640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d0138340 con 0x7fa4d014be90 2024-03-25T18:19:23.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.945+0000 7fa4d6348640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d0137a50 con 0x7fa4d00b4d60 2024-03-25T18:19:23.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.945+0000 7fa4d6348640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d0137be0 con 0x7fa4d00b1730 2024-03-25T18:19:23.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.945+0000 7fa4d6348640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:23.950 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.945+0000 7fa4d6348640 10 monclient(hunting): authenticate will time out at 7550.184992s 2024-03-25T18:19:23.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.952+0000 7fa4cf7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa4d00b4d60 0x7fa4d01dab90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:41612/0 (socket says 172.21.15.64:41612) 2024-03-25T18:19:23.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.952+0000 7fa4cf7fe640 1 -- 172.21.15.64:0/2042473810 learned_addr learned my addr 172.21.15.64:0/2042473810 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:19:23.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.953+0000 7fa4ccff9640 10 client.?.objecter ms_handle_connect 0x7fa4d00b4d60 2024-03-25T18:19:23.955 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.953+0000 7fa4ccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:23.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.965+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888663525 0 0) 0x7fa4d0137a50 con 0x7fa4d00b4d60 2024-03-25T18:19:23.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.965+0000 7fa4ccff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:23.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.965+0000 7fa4ccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:23.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.965+0000 7fa4ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:23.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.965+0000 7fa4ccff9640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:19:23.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.965+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4bc004eb0 con 0x7fa4d00b4d60 2024-03-25T18:19:23.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.972+0000 7fa4ccff9640 10 client.?.objecter ms_handle_connect 0x7fa4d014be90 2024-03-25T18:19:23.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.972+0000 7fa4ccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:23.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.979+0000 7fa4ccff9640 10 client.?.objecter ms_handle_connect 0x7fa4d00b1730 2024-03-25T18:19:23.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.979+0000 7fa4ccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:23.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.985+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1242359012 0 0) 0x7fa4d0138340 con 0x7fa4d014be90 2024-03-25T18:19:23.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.985+0000 7fa4ccff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:23.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.985+0000 7fa4ccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:23.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.985+0000 7fa4ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:23.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.985+0000 7fa4ccff9640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:19:23.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.985+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d0137a50 con 0x7fa4d014be90 2024-03-25T18:19:23.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.991+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3567206752 0 0) 0x7fa4bc004eb0 con 0x7fa4d00b4d60 2024-03-25T18:19:23.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.991+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4d0138340 con 0x7fa4d00b4d60 2024-03-25T18:19:23.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.991+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fa4c4001a10 con 0x7fa4d00b4d60 2024-03-25T18:19:23.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.991+0000 7fa4ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:23.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.991+0000 7fa4ccff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:23.993 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.991+0000 7fa4ccff9640 10 monclient(hunting): dump: 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:23.991+0000 7fa4ccff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:24.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.001+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620825178 0 0) 0x7fa4d0137be0 con 0x7fa4d00b1730 2024-03-25T18:19:24.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.001+0000 7fa4ccff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:24.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.001+0000 7fa4ccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:24.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.001+0000 7fa4ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:24.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.001+0000 7fa4ccff9640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:19:24.004 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.001+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4bc004eb0 con 0x7fa4d00b1730 2024-03-25T18:19:24.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.005+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1075415395 0 0) 0x7fa4d0138340 con 0x7fa4d00b4d60 2024-03-25T18:19:24.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.005+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 >> v1:172.21.15.64:6790/0 conn(0x7fa4d00b1730 legacy=0x7fa4d0131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:24.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.005+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 >> v1:172.21.15.64:6789/0 conn(0x7fa4d014be90 legacy=0x7fa4d01e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:24.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.005+0000 7fa4ccff9640 10 monclient: _finish_hunting 0 2024-03-25T18:19:24.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.005+0000 7fa4ccff9640 1 monclient: found mon.b 2024-03-25T18:19:24.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.005+0000 7fa4ccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:24.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.005+0000 7fa4ccff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:19:24.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.005+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d0140380 con 0x7fa4d00b4d60 2024-03-25T18:19:24.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.005+0000 7fa4ccff9640 10 monclient: _finish_auth 0 2024-03-25T18:19:24.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.005+0000 7fa4ccff9640 10 monclient: _check_auth_tickets 2024-03-25T18:19:24.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.005+0000 7fa4ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:24.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.005+0000 7fa4d6348640 5 monclient: authenticate success, global_id 4672 2024-03-25T18:19:24.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.005+0000 7fa4d6348640 10 monclient: _renew_subs 2024-03-25T18:19:24.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.005+0000 7fa4d6348640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:19:24.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.005+0000 7fa4d6348640 1 -- 172.21.15.64:0/2042473810 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d01405d0 con 0x7fa4d00b4d60 2024-03-25T18:19:24.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.006+0000 7fa4d6348640 10 client.4672.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:24.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.006+0000 7fa4d6348640 10 monclient: _renew_subs 2024-03-25T18:19:24.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.006+0000 7fa4d6348640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:19:24.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.006+0000 7fa4d6348640 1 -- 172.21.15.64:0/2042473810 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d0144f20 con 0x7fa4d00b4d60 2024-03-25T18:19:24.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.006+0000 7fa4d6348640 1 librados: init done 2024-03-25T18:19:24.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.006+0000 7fa4d6348640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:24.010 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.007+0000 7fa4d6348640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:24.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.007+0000 7fa4d6348640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:19:24.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.007+0000 7fa4d6348640 1 -- 172.21.15.64:0/2042473810 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d00627b0 con 0x7fa4d00b4d60 2024-03-25T18:19:24.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.278+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4c4001c70 con 0x7fa4d00b4d60 2024-03-25T18:19:24.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.278+0000 7fa4ccff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:24.280 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.278+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fa4c4003a60 con 0x7fa4d00b4d60 2024-03-25T18:19:24.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.278+0000 7fa4ccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:24.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.278+0000 7fa4ccff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:24.281 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.278+0000 7fa4ccff9640 10 monclient: dump: 2024-03-25T18:19:24.281 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:24.281 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:24.281 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:24.281 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:24.281 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:24.281 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:24.281 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:24.281 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:24.281 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:24.282 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:24.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.278+0000 7fa4ce7fc640 4 set_mon_vals no callback set 2024-03-25T18:19:24.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.279+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fa4c4003c80 con 0x7fa4d00b4d60 2024-03-25T18:19:24.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.279+0000 7fa4cffff640 10 monclient: get_auth_request con 0x7fa4bc06de00 auth_method 0 2024-03-25T18:19:24.283 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.280+0000 7fa4ccff9640 10 client.4672.objecter ms_handle_connect 0x7fa4bc06de00 2024-03-25T18:19:24.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.285+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fa4c4054880 con 0x7fa4d00b4d60 2024-03-25T18:19:24.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.285+0000 7fa4ccff9640 10 client.4672.objecter ms_dispatch 0x7fa4d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:24.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.285+0000 7fa4ccff9640 3 client.4672.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:24.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.285+0000 7fa4ccff9640 3 client.4672.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:24.287 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.285+0000 7fa4ccff9640 20 client.4672.objecter dump_active .. 0 homeless 2024-03-25T18:19:24.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.286+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa4c4050390 con 0x7fa4d00b4d60 2024-03-25T18:19:24.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.286+0000 7fa4ccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:24.288 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.286+0000 7fa4ccff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:24.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.525+0000 7fa4d6348640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:24.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.525+0000 7fa4d6348640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:24.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.525+0000 7fa4d6348640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:19:24.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.525+0000 7fa4d6348640 1 -- 172.21.15.64:0/2042473810 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4d0140760 con 0x7fa4d00b4d60 2024-03-25T18:19:24.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.662+0000 7fa4ccff9640 1 -- 172.21.15.64:0/2042473810 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fa4c40522c0 con 0x7fa4d00b4d60 2024-03-25T18:19:24.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.662+0000 7fa4ccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:24.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.662+0000 7fa4ccff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:24.664 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:19:24.664 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:24.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.664+0000 7fa4d6348640 10 librados: watch_flush enter 2024-03-25T18:19:24.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.664+0000 7fa4d6348640 10 librados: watch_flush exit 2024-03-25T18:19:24.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.664+0000 7fa4d6348640 20 client.4672.objecter shutdown clearing up homeless session... 2024-03-25T18:19:24.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.664+0000 7fa4d6348640 10 client.4672.objecter successfully canceled tick 2024-03-25T18:19:24.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.664+0000 7fa4d6348640 1 -- 172.21.15.64:0/2042473810 >> v1:172.21.15.64:6800/2915543638 conn(0x7fa4bc06de00 legacy=0x7fa4bc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:24.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.664+0000 7fa4d6348640 10 monclient: shutdown 2024-03-25T18:19:24.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.664+0000 7fa4d6348640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:24.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.664+0000 7fa4d6348640 1 -- 172.21.15.64:0/2042473810 >> v1:172.21.15.86:6789/0 conn(0x7fa4d00b4d60 legacy=0x7fa4d01dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:24.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.665+0000 7fa4d6348640 1 -- 172.21.15.64:0/2042473810 shutdown_connections 2024-03-25T18:19:24.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.665+0000 7fa4d6348640 1 -- 172.21.15.64:0/2042473810 >> 172.21.15.64:0/2042473810 conn(0x7fa4d00617a0 msgr2=0x7fa4d00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:24.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.665+0000 7fa4d6348640 1 -- 172.21.15.64:0/2042473810 shutdown_connections 2024-03-25T18:19:24.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.665+0000 7fa4d6348640 1 -- 172.21.15.64:0/2042473810 wait complete. 2024-03-25T18:19:24.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:24.665+0000 7fa4d6348640 1 librados: shutdown 2024-03-25T18:19:24.679 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:31.484 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:31.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.598+0000 7fb04f19d640 10 monclient: get_monmap_and_config 2024-03-25T18:19:31.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.598+0000 7fb04f19d640 10 monclient: build_initial_monmap 2024-03-25T18:19:31.599 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.598+0000 7fb04f19d640 10 monclient: monmap: 2024-03-25T18:19:31.599 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:19:31.599 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:31.600 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:19:31.599528+0000 2024-03-25T18:19:31.600 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:19:31.599528+0000 2024-03-25T18:19:31.600 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:31.600 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:31.600 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:19:31.600 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:19:31.600 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:19:31.600 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:31.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.599+0000 7fb04f19d640 1 Processor -- start 2024-03-25T18:19:31.600 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.599+0000 7fb04f19d640 1 -- start start 2024-03-25T18:19:31.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.599+0000 7fb04f19d640 10 monclient: init 2024-03-25T18:19:31.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.600+0000 7fb04f19d640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:31.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.600+0000 7fb04f19d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:31.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.600+0000 7fb04f19d640 10 monclient(hunting): picked mon.noname-c con 0x7fb0480b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:19:31.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.600+0000 7fb04f19d640 10 monclient(hunting): picked mon.noname-a con 0x7fb0480b1730 addr v1:172.21.15.64:6789/0 2024-03-25T18:19:31.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.600+0000 7fb04f19d640 10 monclient(hunting): picked mon.noname-b con 0x7fb04814be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:19:31.601 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.600+0000 7fb04f19d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0480af940 con 0x7fb0480b1730 2024-03-25T18:19:31.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.600+0000 7fb04f19d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0480b0a70 con 0x7fb0480b4d60 2024-03-25T18:19:31.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.600+0000 7fb04f19d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0480b0d50 con 0x7fb04814be90 2024-03-25T18:19:31.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.600+0000 7fb04f19d640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:31.602 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.600+0000 7fb04f19d640 10 monclient(hunting): authenticate will time out at 7557.839292s 2024-03-25T18:19:31.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.609+0000 7fb047fff640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fb0480b1730 0x7fb0480b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:56412/0 (socket says 172.21.15.64:56412) 2024-03-25T18:19:31.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.609+0000 7fb047fff640 1 -- 172.21.15.64:0/3195260472 learned_addr learned my addr 172.21.15.64:0/3195260472 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:19:31.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.628+0000 7fb0477fe640 1 -- 172.21.15.64:0/3195260472 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42744337 0 0) 0x7fb0480b0a70 con 0x7fb0480b4d60 2024-03-25T18:19:31.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.628+0000 7fb0477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:31.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.628+0000 7fb0477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:31.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.628+0000 7fb0477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:31.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.628+0000 7fb0477fe640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:19:31.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.628+0000 7fb0477fe640 1 -- 172.21.15.64:0/3195260472 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb038003520 con 0x7fb0480b4d60 2024-03-25T18:19:31.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.634+0000 7fb0477fe640 1 -- 172.21.15.64:0/3195260472 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027804226 0 0) 0x7fb0480b0d50 con 0x7fb04814be90 2024-03-25T18:19:31.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.634+0000 7fb0477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:31.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.634+0000 7fb0477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:31.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.634+0000 7fb0477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:31.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.634+0000 7fb0477fe640 10 monclient(hunting): my global_id is 4667 2024-03-25T18:19:31.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.634+0000 7fb0477fe640 1 -- 172.21.15.64:0/3195260472 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0480b0a70 con 0x7fb04814be90 2024-03-25T18:19:31.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.642+0000 7fb0477fe640 1 -- 172.21.15.64:0/3195260472 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 6815626 0 0) 0x7fb038003520 con 0x7fb0480b4d60 2024-03-25T18:19:31.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.642+0000 7fb0477fe640 1 -- 172.21.15.64:0/3195260472 >> v1:172.21.15.64:6790/0 conn(0x7fb04814be90 legacy=0x7fb04814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.642+0000 7fb0477fe640 1 -- 172.21.15.64:0/3195260472 >> v1:172.21.15.64:6789/0 conn(0x7fb0480b1730 legacy=0x7fb0480b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.642+0000 7fb0477fe640 10 monclient: _finish_hunting 0 2024-03-25T18:19:31.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.642+0000 7fb0477fe640 1 monclient: found mon.noname-c 2024-03-25T18:19:31.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.642+0000 7fb0477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:31.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.642+0000 7fb0477fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:19:31.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.642+0000 7fb0477fe640 1 -- 172.21.15.64:0/3195260472 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0480b1030 con 0x7fb0480b4d60 2024-03-25T18:19:31.643 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.642+0000 7fb0477fe640 10 monclient: _finish_auth 0 2024-03-25T18:19:31.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.642+0000 7fb0477fe640 10 monclient: _check_auth_tickets 2024-03-25T18:19:31.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.642+0000 7fb0477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:31.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.642+0000 7fb0477fe640 1 -- 172.21.15.64:0/3195260472 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb03c001690 con 0x7fb0480b4d60 2024-03-25T18:19:31.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.642+0000 7fb0477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:31.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.642+0000 7fb0477fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:31.644 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.642+0000 7fb0477fe640 10 monclient: dump: 2024-03-25T18:19:31.644 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:31.644 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:31.644 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:31.644 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:31.644 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:31.644 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:31.644 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:31.644 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:31.644 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:31.644 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:31.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.642+0000 7fb04f19d640 5 monclient: authenticate success, global_id 4675 2024-03-25T18:19:31.645 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.642+0000 7fb04f19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:31.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.648+0000 7fb0477fe640 1 -- 172.21.15.64:0/3195260472 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb03c0017f0 con 0x7fb0480b4d60 2024-03-25T18:19:31.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.648+0000 7fb0477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:31.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.648+0000 7fb0477fe640 1 -- 172.21.15.64:0/3195260472 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb03c001c40 con 0x7fb0480b4d60 2024-03-25T18:19:31.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.648+0000 7fb0477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:31.649 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.648+0000 7fb0477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:31.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.648+0000 7fb0477fe640 10 monclient: dump: 2024-03-25T18:19:31.650 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:31.650 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:31.650 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:31.650 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:31.650 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:31.650 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:31.650 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:31.650 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:31.650 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:31.650 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:31.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.648+0000 7fb04f19d640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:31.650 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.648+0000 7fb04f19d640 4 set_mon_vals no callback set 2024-03-25T18:19:31.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.649+0000 7fb04f19d640 10 monclient: shutdown 2024-03-25T18:19:31.651 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.649+0000 7fb04f19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:31.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.649+0000 7fb04f19d640 1 -- 172.21.15.64:0/3195260472 >> v1:172.21.15.86:6789/0 conn(0x7fb0480b4d60 legacy=0x7fb04813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.649+0000 7fb04f19d640 1 -- 172.21.15.64:0/3195260472 shutdown_connections 2024-03-25T18:19:31.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.649+0000 7fb04f19d640 1 -- 172.21.15.64:0/3195260472 >> 172.21.15.64:0/3195260472 conn(0x7fb0480617a0 msgr2=0x7fb048061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:31.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.649+0000 7fb04f19d640 1 -- 172.21.15.64:0/3195260472 shutdown_connections 2024-03-25T18:19:31.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.649+0000 7fb04f19d640 1 -- 172.21.15.64:0/3195260472 wait complete. 2024-03-25T18:19:31.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.650+0000 7fb04f19d640 10 monclient: build_initial_monmap 2024-03-25T18:19:31.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.650+0000 7fb04f19d640 10 monclient: monmap: 2024-03-25T18:19:31.652 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:19:31.652 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:31.652 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:19:31.652 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:19:31.652 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:31.652 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:19:31.652 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:19:31.652 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:19:31.653 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:19:31.653 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:31.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.650+0000 7fb04f19d640 1 librados: starting msgr at 2024-03-25T18:19:31.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.650+0000 7fb04f19d640 1 librados: starting objecter 2024-03-25T18:19:31.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.651+0000 7fb04f19d640 1 Processor -- start 2024-03-25T18:19:31.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.651+0000 7fb04f19d640 1 -- start start 2024-03-25T18:19:31.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.651+0000 7fb04f19d640 1 librados: setting wanted keys 2024-03-25T18:19:31.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.651+0000 7fb04f19d640 1 librados: calling monclient init 2024-03-25T18:19:31.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.651+0000 7fb04f19d640 10 monclient: init 2024-03-25T18:19:31.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.652+0000 7fb04f19d640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:31.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.652+0000 7fb04f19d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:31.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.652+0000 7fb04f19d640 10 monclient(hunting): picked mon.noname-a con 0x7fb0480b1730 addr v1:172.21.15.64:6789/0 2024-03-25T18:19:31.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.652+0000 7fb04f19d640 10 monclient(hunting): picked mon.noname-b con 0x7fb0480b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:19:31.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.652+0000 7fb04f19d640 10 monclient(hunting): picked mon.noname-c con 0x7fb04814be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:19:31.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.652+0000 7fb04f19d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb048141ce0 con 0x7fb0480b1730 2024-03-25T18:19:31.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.652+0000 7fb04f19d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb048141fc0 con 0x7fb0480b4d60 2024-03-25T18:19:31.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.652+0000 7fb04f19d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04813bc70 con 0x7fb04814be90 2024-03-25T18:19:31.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.652+0000 7fb04f19d640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:31.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.652+0000 7fb04f19d640 10 monclient(hunting): authenticate will time out at 7557.890894s 2024-03-25T18:19:31.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.663+0000 7fb047fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb0480b4d60 0x7fb0481423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:34864/0 (socket says 172.21.15.64:34864) 2024-03-25T18:19:31.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.663+0000 7fb047fff640 1 -- 172.21.15.64:0/806058170 learned_addr learned my addr 172.21.15.64:0/806058170 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:19:31.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.665+0000 7fb0457fa640 10 client.?.objecter ms_handle_connect 0x7fb0480b4d60 2024-03-25T18:19:31.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.665+0000 7fb0457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:31.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.669+0000 7fb0457fa640 10 client.?.objecter ms_handle_connect 0x7fb0480b1730 2024-03-25T18:19:31.669 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.669+0000 7fb0457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:31.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.670+0000 7fb047fff640 1 -- 172.21.15.64:0/806058170 >> v1:172.21.15.86:6789/0 conn(0x7fb0480b4d60 legacy=0x7fb0481423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:19:31.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.670+0000 7fb047fff640 1 -- 172.21.15.64:0/806058170 >> v1:172.21.15.86:6789/0 conn(0x7fb0480b4d60 legacy=0x7fb0481423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:19:31.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.670+0000 7fb047fff640 1 --1- 172.21.15.64:0/806058170 >> v1:172.21.15.86:6789/0 conn(0x7fb0480b4d60 0x7fb0481423e0 :-1 s=OPENED pgs=73872 cs=1 l=1).handle_message read tag failed 2024-03-25T18:19:31.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.670+0000 7fb047fff640 1 --1- 172.21.15.64:0/806058170 >> v1:172.21.15.86:6789/0 conn(0x7fb0480b4d60 0x7fb0481423e0 :-1 s=OPENED pgs=73872 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:19:31.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.670+0000 7fb0457fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.86:6789/0 2024-03-25T18:19:31.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.672+0000 7fb0457fa640 1 -- 172.21.15.64:0/806058170 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138698469 0 0) 0x7fb048141ce0 con 0x7fb0480b1730 2024-03-25T18:19:31.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.672+0000 7fb0457fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:31.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.672+0000 7fb0457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:31.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.672+0000 7fb0457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:31.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.672+0000 7fb0457fa640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:19:31.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.672+0000 7fb0457fa640 1 -- 172.21.15.64:0/806058170 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb02c004eb0 con 0x7fb0480b1730 2024-03-25T18:19:31.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.675+0000 7fb0457fa640 10 client.?.objecter ms_handle_connect 0x7fb04814be90 2024-03-25T18:19:31.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.675+0000 7fb0457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:31.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.685+0000 7fb0457fa640 1 -- 172.21.15.64:0/806058170 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 313426580 0 0) 0x7fb02c004eb0 con 0x7fb0480b1730 2024-03-25T18:19:31.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.685+0000 7fb0457fa640 1 -- 172.21.15.64:0/806058170 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb048141ce0 con 0x7fb0480b1730 2024-03-25T18:19:31.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.685+0000 7fb0457fa640 1 -- 172.21.15.64:0/806058170 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb03c001bc0 con 0x7fb0480b1730 2024-03-25T18:19:31.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.685+0000 7fb0457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:31.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.685+0000 7fb0457fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:31.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.685+0000 7fb0457fa640 10 monclient(hunting): dump: 2024-03-25T18:19:31.687 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:31.687 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:31.687 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:31.687 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:31.687 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:31.687 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:31.687 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:31.687 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:31.687 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:31.687 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:31.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.685+0000 7fb0457fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:31.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb0457fa640 1 -- 172.21.15.64:0/806058170 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 853181078 0 0) 0x7fb048141ce0 con 0x7fb0480b1730 2024-03-25T18:19:31.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb0457fa640 1 -- 172.21.15.64:0/806058170 >> v1:172.21.15.64:6790/0 conn(0x7fb04814be90 legacy=0x7fb048142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.693 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb0457fa640 1 -- 172.21.15.64:0/806058170 >> v1:172.21.15.86:6789/0 conn(0x7fb0480b4d60 legacy=0x7fb0481423e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:19:31.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb0457fa640 10 monclient: _finish_hunting 0 2024-03-25T18:19:31.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb0457fa640 1 monclient: found mon.a 2024-03-25T18:19:31.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb0457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:31.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb0457fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:19:31.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb0457fa640 1 -- 172.21.15.64:0/806058170 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb048140760 con 0x7fb0480b1730 2024-03-25T18:19:31.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb0457fa640 10 monclient: _finish_auth 0 2024-03-25T18:19:31.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb0457fa640 10 monclient: _check_auth_tickets 2024-03-25T18:19:31.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb0457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:31.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb04f19d640 5 monclient: authenticate success, global_id 4671 2024-03-25T18:19:31.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb04f19d640 10 monclient: _renew_subs 2024-03-25T18:19:31.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb04f19d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:19:31.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb04f19d640 1 -- 172.21.15.64:0/806058170 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb048140c80 con 0x7fb0480b1730 2024-03-25T18:19:31.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb04f19d640 10 client.4671.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:31.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb04f19d640 10 monclient: _renew_subs 2024-03-25T18:19:31.694 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb04f19d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:19:31.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb04f19d640 1 -- 172.21.15.64:0/806058170 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb048140380 con 0x7fb0480b1730 2024-03-25T18:19:31.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.692+0000 7fb04f19d640 1 librados: init done 2024-03-25T18:19:31.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.693+0000 7fb04f19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:31.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.693+0000 7fb04f19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:31.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.693+0000 7fb04f19d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:19:31.695 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:31.693+0000 7fb04f19d640 1 -- 172.21.15.64:0/806058170 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0480afbd0 con 0x7fb0480b1730 2024-03-25T18:19:32.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.006+0000 7fb0457fa640 1 -- 172.21.15.64:0/806058170 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb03c003960 con 0x7fb0480b1730 2024-03-25T18:19:32.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.006+0000 7fb0457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:32.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.006+0000 7fb0457fa640 1 -- 172.21.15.64:0/806058170 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb03c003c70 con 0x7fb0480b1730 2024-03-25T18:19:32.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.006+0000 7fb0457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:32.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.006+0000 7fb0457fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:32.007 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.006+0000 7fb0457fa640 10 monclient: dump: 2024-03-25T18:19:32.008 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:32.008 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:32.008 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:32.008 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:32.008 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:32.008 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:32.008 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:32.008 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:32.008 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:32.008 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:32.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.006+0000 7fb046ffd640 4 set_mon_vals no callback set 2024-03-25T18:19:32.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.007+0000 7fb0457fa640 1 -- 172.21.15.64:0/806058170 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fb03c019c40 con 0x7fb0480b1730 2024-03-25T18:19:32.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.007+0000 7fb047fff640 10 monclient: get_auth_request con 0x7fb02c06dd50 auth_method 0 2024-03-25T18:19:32.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.007+0000 7fb0457fa640 10 client.4671.objecter ms_handle_connect 0x7fb02c06dd50 2024-03-25T18:19:32.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.013+0000 7fb0457fa640 1 -- 172.21.15.64:0/806058170 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1440384392 0 0) 0x7fb03c0536c0 con 0x7fb0480b1730 2024-03-25T18:19:32.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.013+0000 7fb0457fa640 10 client.4671.objecter ms_dispatch 0x7fb048000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:32.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.013+0000 7fb0457fa640 3 client.4671.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:32.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.013+0000 7fb0457fa640 3 client.4671.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:32.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.014+0000 7fb0457fa640 20 client.4671.objecter dump_active .. 0 homeless 2024-03-25T18:19:32.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.014+0000 7fb0457fa640 1 -- 172.21.15.64:0/806058170 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb03c04f1d0 con 0x7fb0480b1730 2024-03-25T18:19:32.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.014+0000 7fb0457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:32.015 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.014+0000 7fb0457fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:32.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.269+0000 7fb04f19d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:32.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.269+0000 7fb04f19d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:32.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.269+0000 7fb04f19d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:19:32.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.269+0000 7fb04f19d640 1 -- 172.21.15.64:0/806058170 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0480627b0 con 0x7fb0480b1730 2024-03-25T18:19:32.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.306+0000 7fb0457fa640 1 -- 172.21.15.64:0/806058170 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2057679798) 0x7fb03c051100 con 0x7fb0480b1730 2024-03-25T18:19:32.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.306+0000 7fb0457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:32.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.306+0000 7fb0457fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:32.308 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:19:32.308 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:09:39.439169+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:09:39.439169+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:32.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.309+0000 7fb04f19d640 10 librados: watch_flush enter 2024-03-25T18:19:32.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.309+0000 7fb04f19d640 10 librados: watch_flush exit 2024-03-25T18:19:32.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.309+0000 7fb04f19d640 20 client.4671.objecter shutdown clearing up homeless session... 2024-03-25T18:19:32.310 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.309+0000 7fb04f19d640 10 client.4671.objecter successfully canceled tick 2024-03-25T18:19:32.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.309+0000 7fb04f19d640 1 -- 172.21.15.64:0/806058170 >> v1:172.21.15.64:6800/2915543638 conn(0x7fb02c06dd50 legacy=0x7fb02c070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:32.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.309+0000 7fb04f19d640 10 monclient: shutdown 2024-03-25T18:19:32.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.309+0000 7fb04f19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:32.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.309+0000 7fb04f19d640 1 -- 172.21.15.64:0/806058170 >> v1:172.21.15.64:6789/0 conn(0x7fb0480b1730 legacy=0x7fb0481db200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:32.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.309+0000 7fb04f19d640 1 -- 172.21.15.64:0/806058170 shutdown_connections 2024-03-25T18:19:32.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.309+0000 7fb04f19d640 1 -- 172.21.15.64:0/806058170 >> 172.21.15.64:0/806058170 conn(0x7fb0480617a0 msgr2=0x7fb04813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:32.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.310+0000 7fb04f19d640 1 -- 172.21.15.64:0/806058170 shutdown_connections 2024-03-25T18:19:32.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.310+0000 7fb04f19d640 1 -- 172.21.15.64:0/806058170 wait complete. 2024-03-25T18:19:32.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:32.310+0000 7fb04f19d640 1 librados: shutdown 2024-03-25T18:19:32.323 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:39.129 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:39.242 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.241+0000 7f28c168b640 10 monclient: get_monmap_and_config 2024-03-25T18:19:39.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.241+0000 7f28c168b640 10 monclient: build_initial_monmap 2024-03-25T18:19:39.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.241+0000 7f28c168b640 10 monclient: monmap: 2024-03-25T18:19:39.243 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:19:39.243 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:39.243 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:19:39.242416+0000 2024-03-25T18:19:39.243 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:19:39.242416+0000 2024-03-25T18:19:39.243 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:39.243 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:39.243 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:19:39.243 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:19:39.243 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:19:39.243 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:39.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.242+0000 7f28c168b640 1 Processor -- start 2024-03-25T18:19:39.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.242+0000 7f28c168b640 1 -- start start 2024-03-25T18:19:39.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.242+0000 7f28c168b640 10 monclient: init 2024-03-25T18:19:39.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.243+0000 7f28c168b640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:39.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.243+0000 7f28c168b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:39.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.243+0000 7f28c168b640 10 monclient(hunting): picked mon.noname-b con 0x7f28bc0a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:19:39.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.243+0000 7f28c168b640 10 monclient(hunting): picked mon.noname-a con 0x7f28bc13f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:19:39.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.243+0000 7f28c168b640 10 monclient(hunting): picked mon.noname-c con 0x7f28bc1430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:19:39.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.243+0000 7f28c168b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28bc142520 con 0x7f28bc13f710 2024-03-25T18:19:39.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.243+0000 7f28c168b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28bc0b28a0 con 0x7f28bc1430c0 2024-03-25T18:19:39.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.243+0000 7f28c168b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28bc0b2a30 con 0x7f28bc0a98f0 2024-03-25T18:19:39.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.243+0000 7f28c168b640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:39.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.243+0000 7f28c168b640 10 monclient(hunting): authenticate will time out at 7565.482069s 2024-03-25T18:19:39.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.253+0000 7f28bb7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f28bc1430c0 0x7f28bc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:57800/0 (socket says 172.21.15.64:57800) 2024-03-25T18:19:39.254 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.254+0000 7f28bb7fe640 1 -- 172.21.15.64:0/567229787 learned_addr learned my addr 172.21.15.64:0/567229787 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:19:39.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.265+0000 7f28ba7fc640 1 -- 172.21.15.64:0/567229787 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218511540 0 0) 0x7f28bc142520 con 0x7f28bc13f710 2024-03-25T18:19:39.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.265+0000 7f28ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:39.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.265+0000 7f28ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:39.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.265+0000 7f28ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:39.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.265+0000 7f28ba7fc640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:19:39.267 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.265+0000 7f28ba7fc640 1 -- 172.21.15.64:0/567229787 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28a8003520 con 0x7f28bc13f710 2024-03-25T18:19:39.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.267+0000 7f28ba7fc640 1 -- 172.21.15.64:0/567229787 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455481921 0 0) 0x7f28bc0b28a0 con 0x7f28bc1430c0 2024-03-25T18:19:39.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.267+0000 7f28ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:39.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.267+0000 7f28ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:39.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.267+0000 7f28ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:39.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.267+0000 7f28ba7fc640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:19:39.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.267+0000 7f28ba7fc640 1 -- 172.21.15.64:0/567229787 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28bc142520 con 0x7f28bc1430c0 2024-03-25T18:19:39.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.272+0000 7f28ba7fc640 1 -- 172.21.15.64:0/567229787 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 903742329 0 0) 0x7f28a8003520 con 0x7f28bc13f710 2024-03-25T18:19:39.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.272+0000 7f28ba7fc640 1 -- 172.21.15.64:0/567229787 >> v1:172.21.15.64:6790/0 conn(0x7f28bc0a98f0 legacy=0x7f28bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.272+0000 7f28ba7fc640 1 -- 172.21.15.64:0/567229787 >> v1:172.21.15.86:6789/0 conn(0x7f28bc1430c0 legacy=0x7f28bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.272+0000 7f28ba7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:39.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.272+0000 7f28ba7fc640 1 monclient: found mon.noname-a 2024-03-25T18:19:39.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.272+0000 7f28ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:39.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.272+0000 7f28ba7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:19:39.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.272+0000 7f28ba7fc640 1 -- 172.21.15.64:0/567229787 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28bc0b44c0 con 0x7f28bc13f710 2024-03-25T18:19:39.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.272+0000 7f28ba7fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:39.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.272+0000 7f28ba7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:39.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.272+0000 7f28ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:39.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.272+0000 7f28ba7fc640 1 -- 172.21.15.64:0/567229787 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f28a4001690 con 0x7f28bc13f710 2024-03-25T18:19:39.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.272+0000 7f28ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:39.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.272+0000 7f28ba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:39.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.272+0000 7f28ba7fc640 10 monclient: dump: 2024-03-25T18:19:39.274 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:39.274 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:39.274 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:39.274 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:39.275 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:39.275 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:39.275 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:39.275 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:39.275 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:39.275 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:39.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.272+0000 7f28c168b640 5 monclient: authenticate success, global_id 4674 2024-03-25T18:19:39.275 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.272+0000 7f28c168b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:39.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.306+0000 7f28ba7fc640 1 -- 172.21.15.64:0/567229787 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28a40017f0 con 0x7f28bc13f710 2024-03-25T18:19:39.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.306+0000 7f28ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:39.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.306+0000 7f28ba7fc640 1 -- 172.21.15.64:0/567229787 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f28a4001c00 con 0x7f28bc13f710 2024-03-25T18:19:39.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.306+0000 7f28c168b640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:39.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.306+0000 7f28c168b640 4 set_mon_vals no callback set 2024-03-25T18:19:39.307 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.306+0000 7f28ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:39.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.306+0000 7f28ba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:39.308 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.306+0000 7f28ba7fc640 10 monclient: dump: 2024-03-25T18:19:39.308 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:39.308 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:39.308 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:39.308 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:39.308 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:39.308 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:39.308 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:39.308 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:39.308 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:39.308 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:39.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.307+0000 7f28c168b640 10 monclient: shutdown 2024-03-25T18:19:39.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.307+0000 7f28c168b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:39.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.307+0000 7f28c168b640 1 -- 172.21.15.64:0/567229787 >> v1:172.21.15.64:6789/0 conn(0x7f28bc13f710 legacy=0x7f28bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.307+0000 7f28c168b640 1 -- 172.21.15.64:0/567229787 shutdown_connections 2024-03-25T18:19:39.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.307+0000 7f28c168b640 1 -- 172.21.15.64:0/567229787 >> 172.21.15.64:0/567229787 conn(0x7f28bc0b7ea0 msgr2=0x7f28bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:39.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.307+0000 7f28c168b640 1 -- 172.21.15.64:0/567229787 shutdown_connections 2024-03-25T18:19:39.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.307+0000 7f28c168b640 1 -- 172.21.15.64:0/567229787 wait complete. 2024-03-25T18:19:39.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.308+0000 7f28c168b640 10 monclient: build_initial_monmap 2024-03-25T18:19:39.311 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.308+0000 7f28c168b640 10 monclient: monmap: 2024-03-25T18:19:39.311 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:19:39.311 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:39.312 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:19:39.312 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:19:39.312 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:39.312 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:19:39.312 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:19:39.312 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:19:39.312 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:19:39.312 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:39.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.308+0000 7f28c168b640 1 librados: starting msgr at 2024-03-25T18:19:39.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.308+0000 7f28c168b640 1 librados: starting objecter 2024-03-25T18:19:39.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.308+0000 7f28c168b640 1 Processor -- start 2024-03-25T18:19:39.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.308+0000 7f28c168b640 1 -- start start 2024-03-25T18:19:39.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.308+0000 7f28c168b640 1 librados: setting wanted keys 2024-03-25T18:19:39.312 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.308+0000 7f28c168b640 1 librados: calling monclient init 2024-03-25T18:19:39.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.308+0000 7f28c168b640 10 monclient: init 2024-03-25T18:19:39.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.309+0000 7f28c168b640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:39.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.309+0000 7f28c168b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:39.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.309+0000 7f28c168b640 10 monclient(hunting): picked mon.noname-a con 0x7f28bc0a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:19:39.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.309+0000 7f28c168b640 10 monclient(hunting): picked mon.noname-b con 0x7f28bc13f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:19:39.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.309+0000 7f28c168b640 10 monclient(hunting): picked mon.noname-c con 0x7f28bc1430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:19:39.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.309+0000 7f28c168b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28bc135550 con 0x7f28bc0a98f0 2024-03-25T18:19:39.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.309+0000 7f28c168b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28bc136b50 con 0x7f28bc13f710 2024-03-25T18:19:39.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.309+0000 7f28c168b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28bc136e30 con 0x7f28bc1430c0 2024-03-25T18:19:39.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.309+0000 7f28c168b640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:39.313 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.309+0000 7f28c168b640 10 monclient(hunting): authenticate will time out at 7565.548328s 2024-03-25T18:19:39.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.310+0000 7f28baffd640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f28bc0a98f0 0x7f28bc13b410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:37018/0 (socket says 172.21.15.64:37018) 2024-03-25T18:19:39.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.310+0000 7f28baffd640 1 -- 172.21.15.64:0/1498812818 learned_addr learned my addr 172.21.15.64:0/1498812818 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:19:39.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.313+0000 7f28b37fe640 10 client.?.objecter ms_handle_connect 0x7f28bc0a98f0 2024-03-25T18:19:39.314 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.313+0000 7f28b37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:39.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.317+0000 7f28b37fe640 10 client.?.objecter ms_handle_connect 0x7f28bc1430c0 2024-03-25T18:19:39.318 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.317+0000 7f28b37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:39.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.319+0000 7f28b37fe640 10 client.?.objecter ms_handle_connect 0x7f28bc13f710 2024-03-25T18:19:39.320 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.319+0000 7f28b37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:39.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.324+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439145443 0 0) 0x7f28bc136b50 con 0x7f28bc13f710 2024-03-25T18:19:39.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.324+0000 7f28b37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:39.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.324+0000 7f28b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:39.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.324+0000 7f28b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:39.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.324+0000 7f28b37fe640 10 monclient(hunting): my global_id is 4681 2024-03-25T18:19:39.325 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.324+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f289c004eb0 con 0x7f28bc13f710 2024-03-25T18:19:39.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.330+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256471295 0 0) 0x7f28bc135550 con 0x7f28bc0a98f0 2024-03-25T18:19:39.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.330+0000 7f28b37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:39.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.330+0000 7f28b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:39.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.330+0000 7f28b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:39.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.330+0000 7f28b37fe640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:19:39.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.330+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28bc136b50 con 0x7f28bc0a98f0 2024-03-25T18:19:39.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.336+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4196250718 0 0) 0x7f289c004eb0 con 0x7f28bc13f710 2024-03-25T18:19:39.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.336+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28bc135550 con 0x7f28bc13f710 2024-03-25T18:19:39.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.336+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f28a4001bc0 con 0x7f28bc13f710 2024-03-25T18:19:39.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.336+0000 7f28b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:39.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.336+0000 7f28b37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:39.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.336+0000 7f28b37fe640 10 monclient(hunting): dump: 2024-03-25T18:19:39.338 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:39.338 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:39.339 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:39.339 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:39.339 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:39.339 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:39.339 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:39.339 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:39.339 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:39.339 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:39.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.337+0000 7f28b37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:39.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.339+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1204862570 0 0) 0x7f28bc136b50 con 0x7f28bc0a98f0 2024-03-25T18:19:39.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.339+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f289c004eb0 con 0x7f28bc0a98f0 2024-03-25T18:19:39.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.339+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f28ac001970 con 0x7f28bc0a98f0 2024-03-25T18:19:39.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.339+0000 7f28b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:39.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.339+0000 7f28b37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:39.340 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.339+0000 7f28b37fe640 10 monclient(hunting): dump: 2024-03-25T18:19:39.341 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:39.341 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:39.341 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:39.341 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:39.341 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:39.341 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:39.341 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:39.341 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:39.341 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:39.341 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:39.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.340+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718948934 0 0) 0x7f28bc136e30 con 0x7f28bc1430c0 2024-03-25T18:19:39.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.340+0000 7f28b37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:39.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.340+0000 7f28b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:39.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.340+0000 7f28b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:39.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.340+0000 7f28b37fe640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:19:39.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.340+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28bc136b50 con 0x7f28bc1430c0 2024-03-25T18:19:39.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2273304516 0 0) 0x7f289c004eb0 con 0x7f28bc0a98f0 2024-03-25T18:19:39.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 >> v1:172.21.15.64:6790/0 conn(0x7f28bc1430c0 legacy=0x7f28bc1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 >> v1:172.21.15.86:6789/0 conn(0x7f28bc13f710 legacy=0x7f28bc1375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28b37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:19:39.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28b37fe640 1 monclient: found mon.a 2024-03-25T18:19:39.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:39.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28b37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:19:39.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28bc1357f0 con 0x7f28bc0a98f0 2024-03-25T18:19:39.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28b37fe640 10 monclient: _finish_auth 0 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28b37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28c168b640 5 monclient: authenticate success, global_id 4677 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28c168b640 10 monclient: _renew_subs 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28c168b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28c168b640 1 -- 172.21.15.64:0/1498812818 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28bc137cb0 con 0x7f28bc0a98f0 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28c168b640 10 client.4677.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28c168b640 10 monclient: _renew_subs 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28c168b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28c168b640 1 -- 172.21.15.64:0/1498812818 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28bc1352e0 con 0x7f28bc0a98f0 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.345+0000 7f28c168b640 1 librados: init done 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.346+0000 7f28c168b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:39.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.346+0000 7f28c168b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:39.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.346+0000 7f28c168b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:19:39.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.346+0000 7f28c168b640 1 -- 172.21.15.64:0/1498812818 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28bc0627b0 con 0x7f28bc0a98f0 2024-03-25T18:19:39.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.670+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28ac003610 con 0x7f28bc0a98f0 2024-03-25T18:19:39.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.670+0000 7f28b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:39.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.670+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f28ac0039a0 con 0x7f28bc0a98f0 2024-03-25T18:19:39.671 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.670+0000 7f28b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:39.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.670+0000 7f28b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:39.672 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.670+0000 7f28b37fe640 10 monclient: dump: 2024-03-25T18:19:39.672 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:39.672 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:39.672 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:39.672 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:39.672 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:39.672 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:39.672 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:39.672 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:39.672 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:39.672 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:39.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.670+0000 7f28b9ffb640 4 set_mon_vals no callback set 2024-03-25T18:19:39.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.671+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f28ac003bc0 con 0x7f28bc0a98f0 2024-03-25T18:19:39.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.671+0000 7f28b3fff640 10 monclient: get_auth_request con 0x7f289c06df40 auth_method 0 2024-03-25T18:19:39.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.671+0000 7f28b37fe640 10 client.4677.objecter ms_handle_connect 0x7f289c06df40 2024-03-25T18:19:39.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.678+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4227240091 0 0) 0x7f28ac054790 con 0x7f28bc0a98f0 2024-03-25T18:19:39.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.678+0000 7f28b37fe640 10 client.4677.objecter ms_dispatch 0x7f28bc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:39.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.678+0000 7f28b37fe640 3 client.4677.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:39.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.678+0000 7f28b37fe640 3 client.4677.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:39.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.678+0000 7f28b37fe640 20 client.4677.objecter dump_active .. 0 homeless 2024-03-25T18:19:39.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.678+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f28ac0502a0 con 0x7f28bc0a98f0 2024-03-25T18:19:39.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.678+0000 7f28b37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:39.679 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.678+0000 7f28b37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:39.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.919+0000 7f28c168b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:39.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.919+0000 7f28c168b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:39.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.919+0000 7f28c168b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:19:39.920 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:39.919+0000 7f28c168b640 1 -- 172.21.15.64:0/1498812818 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28bc074cc0 con 0x7f28bc0a98f0 2024-03-25T18:19:40.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:40.018+0000 7f28b37fe640 1 -- 172.21.15.64:0/1498812818 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 991200065) 0x7f28ac0521d0 con 0x7f28bc0a98f0 2024-03-25T18:19:40.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:40.018+0000 7f28b37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:40.019 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:40.018+0000 7f28b37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:40.019 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:19:40.019 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:19:38.421162+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:38.421162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:40.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:40.020+0000 7f28c168b640 10 librados: watch_flush enter 2024-03-25T18:19:40.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:40.020+0000 7f28c168b640 10 librados: watch_flush exit 2024-03-25T18:19:40.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:40.020+0000 7f28c168b640 20 client.4677.objecter shutdown clearing up homeless session... 2024-03-25T18:19:40.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:40.020+0000 7f28c168b640 10 client.4677.objecter successfully canceled tick 2024-03-25T18:19:40.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:40.020+0000 7f28c168b640 1 -- 172.21.15.64:0/1498812818 >> v1:172.21.15.64:6800/2915543638 conn(0x7f289c06df40 legacy=0x7f289c070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:40.022 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:40.020+0000 7f28c168b640 10 monclient: shutdown 2024-03-25T18:19:40.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:40.020+0000 7f28c168b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:40.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:40.020+0000 7f28c168b640 1 -- 172.21.15.64:0/1498812818 >> v1:172.21.15.64:6789/0 conn(0x7f28bc0a98f0 legacy=0x7f28bc13b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:40.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:40.020+0000 7f28c168b640 1 -- 172.21.15.64:0/1498812818 shutdown_connections 2024-03-25T18:19:40.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:40.020+0000 7f28c168b640 1 -- 172.21.15.64:0/1498812818 >> 172.21.15.64:0/1498812818 conn(0x7f28bc0b7ea0 msgr2=0x7f28bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:40.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:40.020+0000 7f28c168b640 1 -- 172.21.15.64:0/1498812818 shutdown_connections 2024-03-25T18:19:40.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:40.020+0000 7f28c168b640 1 -- 172.21.15.64:0/1498812818 wait complete. 2024-03-25T18:19:40.023 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:40.020+0000 7f28c168b640 1 librados: shutdown 2024-03-25T18:19:40.034 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:43.504 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:10:26.617 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:43.555 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:10:26.668 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:43.601 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:10:26.713 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:43.609 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:10:26.687 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:43.614 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:10:26.726 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:43.617 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:10:26.695 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:43.625 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:10:26.737 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:43.647 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:10:26.725 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:43.657 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:10:26.770 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:46.841 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:46.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.942+0000 7fd2a619f640 10 monclient: get_monmap_and_config 2024-03-25T18:19:46.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.942+0000 7fd2a619f640 10 monclient: build_initial_monmap 2024-03-25T18:19:46.943 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.942+0000 7fd2a619f640 10 monclient: monmap: 2024-03-25T18:19:46.943 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:19:46.943 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:46.943 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:19:46.943119+0000 2024-03-25T18:19:46.944 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:19:46.943119+0000 2024-03-25T18:19:46.944 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:46.944 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:46.944 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:19:46.944 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:19:46.944 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:19:46.944 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:46.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.942+0000 7fd2a619f640 1 Processor -- start 2024-03-25T18:19:46.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.943+0000 7fd2a619f640 1 -- start start 2024-03-25T18:19:46.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.943+0000 7fd2a619f640 10 monclient: init 2024-03-25T18:19:46.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.943+0000 7fd2a619f640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:46.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.943+0000 7fd2a619f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:46.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.943+0000 7fd2a619f640 10 monclient(hunting): picked mon.noname-a con 0x7fd2a00b48e0 addr v1:172.21.15.64:6789/0 2024-03-25T18:19:46.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.943+0000 7fd2a619f640 10 monclient(hunting): picked mon.noname-b con 0x7fd2a013fc60 addr v1:172.21.15.64:6790/0 2024-03-25T18:19:46.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.943+0000 7fd2a619f640 10 monclient(hunting): picked mon.noname-c con 0x7fd2a00b3f90 addr v1:172.21.15.86:6789/0 2024-03-25T18:19:46.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.943+0000 7fd2a619f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a00a9d10 con 0x7fd2a00b48e0 2024-03-25T18:19:46.945 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.943+0000 7fd2a619f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a00ab000 con 0x7fd2a00b3f90 2024-03-25T18:19:46.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.944+0000 7fd2a619f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a00a9680 con 0x7fd2a013fc60 2024-03-25T18:19:46.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.944+0000 7fd2a619f640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:46.946 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.944+0000 7fd2a619f640 10 monclient(hunting): authenticate will time out at 7573.182484s 2024-03-25T18:19:46.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.953+0000 7fd2a499c640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fd2a013fc60 0x7fd2a0145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:35578/0 (socket says 172.21.15.64:35578) 2024-03-25T18:19:46.954 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.953+0000 7fd2a499c640 1 -- 172.21.15.64:0/910647727 learned_addr learned my addr 172.21.15.64:0/910647727 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:19:46.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.961+0000 7fd297fff640 1 -- 172.21.15.64:0/910647727 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951798251 0 0) 0x7fd2a00a9680 con 0x7fd2a013fc60 2024-03-25T18:19:46.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.961+0000 7fd297fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:46.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.961+0000 7fd297fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:46.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.961+0000 7fd297fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:46.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.961+0000 7fd297fff640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:19:46.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.961+0000 7fd297fff640 1 -- 172.21.15.64:0/910647727 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd28c003520 con 0x7fd2a013fc60 2024-03-25T18:19:46.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.971+0000 7fd297fff640 1 -- 172.21.15.64:0/910647727 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 865295000 0 0) 0x7fd28c003520 con 0x7fd2a013fc60 2024-03-25T18:19:46.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.971+0000 7fd297fff640 1 -- 172.21.15.64:0/910647727 >> v1:172.21.15.86:6789/0 conn(0x7fd2a00b3f90 legacy=0x7fd2a014dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:46.972 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.971+0000 7fd297fff640 1 -- 172.21.15.64:0/910647727 >> v1:172.21.15.64:6789/0 conn(0x7fd2a00b48e0 legacy=0x7fd2a013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.971+0000 7fd297fff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.971+0000 7fd297fff640 1 monclient: found mon.noname-b 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.971+0000 7fd297fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.971+0000 7fd297fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.971+0000 7fd297fff640 1 -- 172.21.15.64:0/910647727 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a00a9810 con 0x7fd2a013fc60 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.971+0000 7fd297fff640 10 monclient: _finish_auth 0 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.971+0000 7fd297fff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.971+0000 7fd297fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.971+0000 7fd297fff640 1 -- 172.21.15.64:0/910647727 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fd290001690 con 0x7fd2a013fc60 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.971+0000 7fd297fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.971+0000 7fd297fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.971+0000 7fd297fff640 10 monclient: dump: 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.971+0000 7fd2a619f640 5 monclient: authenticate success, global_id 4679 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.971+0000 7fd2a619f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.980+0000 7fd297fff640 1 -- 172.21.15.64:0/910647727 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd290001860 con 0x7fd2a013fc60 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.980+0000 7fd297fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.980+0000 7fd297fff640 1 -- 172.21.15.64:0/910647727 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fd290001cb0 con 0x7fd2a013fc60 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.980+0000 7fd2a619f640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:46.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.980+0000 7fd2a619f640 4 set_mon_vals no callback set 2024-03-25T18:19:46.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.980+0000 7fd297fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:46.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.980+0000 7fd297fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:46.982 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.980+0000 7fd297fff640 10 monclient: dump: 2024-03-25T18:19:46.982 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:46.982 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:46.982 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:46.982 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:46.982 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:46.982 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:46.982 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:46.982 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:46.982 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:46.982 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:46.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.981+0000 7fd2a619f640 10 monclient: shutdown 2024-03-25T18:19:46.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.981+0000 7fd2a619f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:46.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.981+0000 7fd2a619f640 1 -- 172.21.15.64:0/910647727 >> v1:172.21.15.64:6790/0 conn(0x7fd2a013fc60 legacy=0x7fd2a0145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:46.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.981+0000 7fd2a619f640 1 -- 172.21.15.64:0/910647727 shutdown_connections 2024-03-25T18:19:46.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.981+0000 7fd2a619f640 1 -- 172.21.15.64:0/910647727 >> 172.21.15.64:0/910647727 conn(0x7fd2a00615c0 msgr2=0x7fd2a00619d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:46.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.981+0000 7fd2a619f640 1 -- 172.21.15.64:0/910647727 shutdown_connections 2024-03-25T18:19:46.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.981+0000 7fd2a619f640 1 -- 172.21.15.64:0/910647727 wait complete. 2024-03-25T18:19:46.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.982+0000 7fd2a619f640 10 monclient: build_initial_monmap 2024-03-25T18:19:46.984 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.982+0000 7fd2a619f640 10 monclient: monmap: 2024-03-25T18:19:46.984 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:19:46.984 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:46.984 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:19:46.985 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:19:46.985 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:46.985 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:19:46.985 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:19:46.985 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:19:46.985 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:19:46.985 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:46.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.983+0000 7fd2a619f640 1 librados: starting msgr at 2024-03-25T18:19:46.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.983+0000 7fd2a619f640 1 librados: starting objecter 2024-03-25T18:19:46.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.983+0000 7fd2a619f640 1 Processor -- start 2024-03-25T18:19:46.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.983+0000 7fd2a619f640 1 -- start start 2024-03-25T18:19:46.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.983+0000 7fd2a619f640 1 librados: setting wanted keys 2024-03-25T18:19:46.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.983+0000 7fd2a619f640 1 librados: calling monclient init 2024-03-25T18:19:46.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.983+0000 7fd2a619f640 10 monclient: init 2024-03-25T18:19:46.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.983+0000 7fd2a619f640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:46.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.983+0000 7fd2a619f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:46.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.983+0000 7fd2a619f640 10 monclient(hunting): picked mon.noname-b con 0x7fd2a00b3f90 addr v1:172.21.15.86:6789/0 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.983+0000 7fd2a619f640 10 monclient(hunting): picked mon.noname-a con 0x7fd2a00b48e0 addr v1:172.21.15.64:6789/0 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.984+0000 7fd2a619f640 10 monclient(hunting): picked mon.noname-c con 0x7fd2a013fc60 addr v1:172.21.15.64:6790/0 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.984+0000 7fd2a619f640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a00afec0 con 0x7fd2a00b48e0 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.984+0000 7fd2a619f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a00b0130 con 0x7fd2a00b3f90 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.984+0000 7fd2a619f640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a01da600 con 0x7fd2a013fc60 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.984+0000 7fd2a619f640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.984+0000 7fd2a619f640 10 monclient(hunting): authenticate will time out at 7573.222635s 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.986+0000 7fd2a519d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd2a00b3f90 0x7fd2a00b0960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:53752/0 (socket says 172.21.15.64:53752) 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.986+0000 7fd2a519d640 1 -- 172.21.15.64:0/736888558 learned_addr learned my addr 172.21.15.64:0/736888558 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:19:46.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.987+0000 7fd295ffb640 10 client.?.objecter ms_handle_connect 0x7fd2a00b3f90 2024-03-25T18:19:46.988 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.987+0000 7fd295ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:47.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.999+0000 7fd295ffb640 10 client.?.objecter ms_handle_connect 0x7fd2a00b48e0 2024-03-25T18:19:47.000 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:46.999+0000 7fd295ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:47.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.000+0000 7fd295ffb640 10 client.?.objecter ms_handle_connect 0x7fd2a013fc60 2024-03-25T18:19:47.001 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.000+0000 7fd295ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:47.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.002+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771748857 0 0) 0x7fd2a00b0130 con 0x7fd2a00b3f90 2024-03-25T18:19:47.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.002+0000 7fd295ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:47.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.002+0000 7fd295ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:47.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.002+0000 7fd295ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:47.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.002+0000 7fd295ffb640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:19:47.003 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.002+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd284004eb0 con 0x7fd2a00b3f90 2024-03-25T18:19:47.013 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.012+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770751837 0 0) 0x7fd2a01da600 con 0x7fd2a013fc60 2024-03-25T18:19:47.013 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.012+0000 7fd295ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:47.013 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.012+0000 7fd295ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:47.013 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.012+0000 7fd295ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:47.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.012+0000 7fd295ffb640 10 monclient(hunting): my global_id is 4682 2024-03-25T18:19:47.014 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.012+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2a00b0130 con 0x7fd2a013fc60 2024-03-25T18:19:47.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.015+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174366213 0 0) 0x7fd2a00afec0 con 0x7fd2a00b48e0 2024-03-25T18:19:47.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.015+0000 7fd295ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:47.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.015+0000 7fd295ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:47.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.015+0000 7fd295ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:47.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.015+0000 7fd295ffb640 10 monclient(hunting): my global_id is 4680 2024-03-25T18:19:47.016 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.015+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2a01da600 con 0x7fd2a00b48e0 2024-03-25T18:19:47.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.018+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3022273143 0 0) 0x7fd284004eb0 con 0x7fd2a00b3f90 2024-03-25T18:19:47.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.018+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2a00afec0 con 0x7fd2a00b3f90 2024-03-25T18:19:47.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.018+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fd29c004080 con 0x7fd2a00b3f90 2024-03-25T18:19:47.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.018+0000 7fd295ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.018+0000 7fd295ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:47.020 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.018+0000 7fd295ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:47.020 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:47.020 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:47.020 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:47.020 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:47.020 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.020 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:47.020 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:47.020 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:47.021 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:47.021 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:47.021 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.018+0000 7fd295ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:47.025 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.024+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1457762964 0 0) 0x7fd2a00b0130 con 0x7fd2a013fc60 2024-03-25T18:19:47.026 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.024+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd284004eb0 con 0x7fd2a013fc60 2024-03-25T18:19:47.026 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.024+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fd298001bb0 con 0x7fd2a013fc60 2024-03-25T18:19:47.026 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.024+0000 7fd295ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.026 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.024+0000 7fd295ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:47.026 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.024+0000 7fd295ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:47.026 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:47.026 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:47.026 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:47.026 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:47.026 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.026 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:47.026 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:47.026 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:47.026 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:47.026 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:47.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.026+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1248549268 0 0) 0x7fd2a01da600 con 0x7fd2a00b48e0 2024-03-25T18:19:47.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.026+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2a00b0130 con 0x7fd2a00b48e0 2024-03-25T18:19:47.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.026+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fd290001b70 con 0x7fd2a00b48e0 2024-03-25T18:19:47.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.026+0000 7fd295ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.026+0000 7fd295ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:47.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.026+0000 7fd295ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:47.027 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:47.027 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:47.028 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:47.028 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:47.028 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.028 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:47.028 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:47.028 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:47.028 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:47.028 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:47.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.033+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2489554834 0 0) 0x7fd2a00afec0 con 0x7fd2a00b3f90 2024-03-25T18:19:47.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 >> v1:172.21.15.64:6790/0 conn(0x7fd2a013fc60 legacy=0x7fd2a01e1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 >> v1:172.21.15.64:6789/0 conn(0x7fd2a00b48e0 legacy=0x7fd2a00b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd295ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:47.035 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd295ffb640 1 monclient: found mon.b 2024-03-25T18:19:47.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd295ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:47.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd295ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:19:47.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a00ae950 con 0x7fd2a00b3f90 2024-03-25T18:19:47.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd295ffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:47.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd295ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:47.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd295ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:47.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd2a619f640 5 monclient: authenticate success, global_id 4687 2024-03-25T18:19:47.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd2a619f640 10 monclient: _renew_subs 2024-03-25T18:19:47.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd2a619f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:19:47.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd2a619f640 1 -- 172.21.15.64:0/736888558 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a00aec00 con 0x7fd2a00b3f90 2024-03-25T18:19:47.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd2a619f640 10 client.4687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:47.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd2a619f640 10 monclient: _renew_subs 2024-03-25T18:19:47.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd2a619f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:19:47.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd2a619f640 1 -- 172.21.15.64:0/736888558 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a00adeb0 con 0x7fd2a00b3f90 2024-03-25T18:19:47.036 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.034+0000 7fd2a619f640 1 librados: init done 2024-03-25T18:19:47.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.035+0000 7fd2a619f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:47.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.035+0000 7fd2a619f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:47.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.035+0000 7fd2a619f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:19:47.037 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.035+0000 7fd2a619f640 1 -- 172.21.15.64:0/736888558 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a00627b0 con 0x7fd2a00b3f90 2024-03-25T18:19:47.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.329+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd29c000ed0 con 0x7fd2a00b3f90 2024-03-25T18:19:47.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.329+0000 7fd295ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:47.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.329+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fd29c0012a0 con 0x7fd2a00b3f90 2024-03-25T18:19:47.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.329+0000 7fd295ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.329+0000 7fd295ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:47.330 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.329+0000 7fd295ffb640 10 monclient: dump: 2024-03-25T18:19:47.330 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:47.330 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:47.330 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:47.331 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:47.331 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.331 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:47.331 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:47.331 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:47.331 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:47.331 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:47.331 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.329+0000 7fd2977fe640 4 set_mon_vals no callback set 2024-03-25T18:19:47.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.330+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fd29c0014c0 con 0x7fd2a00b3f90 2024-03-25T18:19:47.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.330+0000 7fd2a499c640 10 monclient: get_auth_request con 0x7fd28406e0f0 auth_method 0 2024-03-25T18:19:47.332 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.331+0000 7fd295ffb640 10 client.4687.objecter ms_handle_connect 0x7fd28406e0f0 2024-03-25T18:19:47.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.335+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4227240091 0 0) 0x7fd29c0547b0 con 0x7fd2a00b3f90 2024-03-25T18:19:47.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.335+0000 7fd295ffb640 10 client.4687.objecter ms_dispatch 0x7fd2a0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:47.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.335+0000 7fd295ffb640 3 client.4687.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:47.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.335+0000 7fd295ffb640 3 client.4687.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:47.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.335+0000 7fd295ffb640 20 client.4687.objecter dump_active .. 0 homeless 2024-03-25T18:19:47.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.335+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd29c0502c0 con 0x7fd2a00b3f90 2024-03-25T18:19:47.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.335+0000 7fd295ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:47.336 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.335+0000 7fd295ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:47.596 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.595+0000 7fd2a619f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:47.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.595+0000 7fd2a619f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:47.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.595+0000 7fd2a619f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:19:47.597 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.595+0000 7fd2a619f640 1 -- 172.21.15.64:0/736888558 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2a00aed90 con 0x7fd2a00b3f90 2024-03-25T18:19:47.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.625+0000 7fd295ffb640 1 -- 172.21.15.64:0/736888558 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 991200065) 0x7fd29c0521f0 con 0x7fd2a00b3f90 2024-03-25T18:19:47.626 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.625+0000 7fd295ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:47.627 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.625+0000 7fd295ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:47.627 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:19:47.627 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:19:38.421162+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:38.421162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:47.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.627+0000 7fd2a619f640 10 librados: watch_flush enter 2024-03-25T18:19:47.629 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.628+0000 7fd2a619f640 10 librados: watch_flush exit 2024-03-25T18:19:47.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.628+0000 7fd2a619f640 20 client.4687.objecter shutdown clearing up homeless session... 2024-03-25T18:19:47.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.628+0000 7fd2a619f640 10 client.4687.objecter successfully canceled tick 2024-03-25T18:19:47.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.628+0000 7fd2a619f640 1 -- 172.21.15.64:0/736888558 >> v1:172.21.15.64:6800/2915543638 conn(0x7fd28406e0f0 legacy=0x7fd2840705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.628+0000 7fd2a619f640 10 monclient: shutdown 2024-03-25T18:19:47.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.628+0000 7fd2a619f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:47.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.628+0000 7fd2a619f640 1 -- 172.21.15.64:0/736888558 >> v1:172.21.15.86:6789/0 conn(0x7fd2a00b3f90 legacy=0x7fd2a00b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.628+0000 7fd2a619f640 1 -- 172.21.15.64:0/736888558 shutdown_connections 2024-03-25T18:19:47.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.628+0000 7fd2a619f640 1 -- 172.21.15.64:0/736888558 >> 172.21.15.64:0/736888558 conn(0x7fd2a00615c0 msgr2=0x7fd2a00b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:47.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.628+0000 7fd2a619f640 1 -- 172.21.15.64:0/736888558 shutdown_connections 2024-03-25T18:19:47.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.628+0000 7fd2a619f640 1 -- 172.21.15.64:0/736888558 wait complete. 2024-03-25T18:19:47.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:47.628+0000 7fd2a619f640 1 librados: shutdown 2024-03-25T18:19:47.641 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:54.447 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:54.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.561+0000 7f6674d72640 10 monclient: get_monmap_and_config 2024-03-25T18:19:54.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.561+0000 7f6674d72640 10 monclient: build_initial_monmap 2024-03-25T18:19:54.562 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.561+0000 7f6674d72640 10 monclient: monmap: 2024-03-25T18:19:54.562 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:19:54.562 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:54.562 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:19:54.562279+0000 2024-03-25T18:19:54.563 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:19:54.562279+0000 2024-03-25T18:19:54.563 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:54.563 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:54.563 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:19:54.563 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:19:54.563 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:19:54.563 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:54.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.562+0000 7f6674d72640 1 Processor -- start 2024-03-25T18:19:54.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.562+0000 7f6674d72640 1 -- start start 2024-03-25T18:19:54.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.562+0000 7f6674d72640 10 monclient: init 2024-03-25T18:19:54.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.563+0000 7f6674d72640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:54.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.563+0000 7f6674d72640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:54.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.563+0000 7f6674d72640 10 monclient(hunting): picked mon.noname-c con 0x7f66700a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:19:54.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.563+0000 7f6674d72640 10 monclient(hunting): picked mon.noname-b con 0x7f667013f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:19:54.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.563+0000 7f6674d72640 10 monclient(hunting): picked mon.noname-a con 0x7f66701430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:19:54.564 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.563+0000 7f6674d72640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6670142520 con 0x7f66701430c0 2024-03-25T18:19:54.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.563+0000 7f6674d72640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66700b28a0 con 0x7f66700a98f0 2024-03-25T18:19:54.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.563+0000 7f6674d72640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66700b2a30 con 0x7f667013f710 2024-03-25T18:19:54.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.563+0000 7f6674d72640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:54.565 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.563+0000 7f6674d72640 10 monclient(hunting): authenticate will time out at 7580.801693s 2024-03-25T18:19:54.573 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.572+0000 7f666e59c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f66700a98f0 0x7f66700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:58460/0 (socket says 172.21.15.64:58460) 2024-03-25T18:19:54.573 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.572+0000 7f666e59c640 1 -- 172.21.15.64:0/510312666 learned_addr learned my addr 172.21.15.64:0/510312666 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:19:54.583 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.583+0000 7f666d59a640 1 -- 172.21.15.64:0/510312666 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294553692 0 0) 0x7f6670142520 con 0x7f66701430c0 2024-03-25T18:19:54.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.583+0000 7f666d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.583+0000 7f666d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.583+0000 7f666d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.583+0000 7f666d59a640 10 monclient(hunting): my global_id is 4683 2024-03-25T18:19:54.584 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.583+0000 7f666d59a640 1 -- 172.21.15.64:0/510312666 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6658003520 con 0x7f66701430c0 2024-03-25T18:19:54.592 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.591+0000 7f666d59a640 1 -- 172.21.15.64:0/510312666 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2893317764 0 0) 0x7f6658003520 con 0x7f66701430c0 2024-03-25T18:19:54.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.591+0000 7f666d59a640 1 -- 172.21.15.64:0/510312666 >> v1:172.21.15.64:6790/0 conn(0x7f667013f710 legacy=0x7f66701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.591+0000 7f666d59a640 1 -- 172.21.15.64:0/510312666 >> v1:172.21.15.86:6789/0 conn(0x7f66700a98f0 legacy=0x7f66700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.591+0000 7f666d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:19:54.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.591+0000 7f666d59a640 1 monclient: found mon.noname-a 2024-03-25T18:19:54.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.591+0000 7f666d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:54.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.591+0000 7f666d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:19:54.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.591+0000 7f666d59a640 1 -- 172.21.15.64:0/510312666 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66700b44c0 con 0x7f66701430c0 2024-03-25T18:19:54.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.591+0000 7f666d59a640 10 monclient: _finish_auth 0 2024-03-25T18:19:54.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.591+0000 7f666d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:19:54.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.591+0000 7f666d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:54.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.591+0000 7f666d59a640 1 -- 172.21.15.64:0/510312666 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f6668001a10 con 0x7f66701430c0 2024-03-25T18:19:54.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.591+0000 7f666d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.591+0000 7f666d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:54.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.591+0000 7f666d59a640 10 monclient: dump: 2024-03-25T18:19:54.594 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:54.594 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:54.594 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:54.594 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:54.594 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.594 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:54.594 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:54.594 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:54.594 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:54.594 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:54.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.591+0000 7f6674d72640 5 monclient: authenticate success, global_id 4683 2024-03-25T18:19:54.594 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.591+0000 7f6674d72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:54.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.608+0000 7f666d59a640 1 -- 172.21.15.64:0/510312666 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6668001b70 con 0x7f66701430c0 2024-03-25T18:19:54.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.608+0000 7f666d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:54.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.608+0000 7f666d59a640 1 -- 172.21.15.64:0/510312666 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f6668001f80 con 0x7f66701430c0 2024-03-25T18:19:54.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.608+0000 7f666d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.608+0000 7f666d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:54.609 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.608+0000 7f666d59a640 10 monclient: dump: 2024-03-25T18:19:54.609 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:54.609 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:54.609 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:54.610 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:54.610 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.610 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:54.610 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:54.610 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:54.610 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:54.610 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:54.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.608+0000 7f6674d72640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:54.610 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.608+0000 7f6674d72640 4 set_mon_vals no callback set 2024-03-25T18:19:54.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.608+0000 7f6674d72640 10 monclient: shutdown 2024-03-25T18:19:54.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.608+0000 7f6674d72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:54.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.608+0000 7f6674d72640 1 -- 172.21.15.64:0/510312666 >> v1:172.21.15.64:6789/0 conn(0x7f66701430c0 legacy=0x7f667014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.609+0000 7f6674d72640 1 -- 172.21.15.64:0/510312666 shutdown_connections 2024-03-25T18:19:54.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.609+0000 7f6674d72640 1 -- 172.21.15.64:0/510312666 >> 172.21.15.64:0/510312666 conn(0x7f66700b7ea0 msgr2=0x7f66700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:54.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.609+0000 7f6674d72640 1 -- 172.21.15.64:0/510312666 shutdown_connections 2024-03-25T18:19:54.612 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.609+0000 7f6674d72640 1 -- 172.21.15.64:0/510312666 wait complete. 2024-03-25T18:19:54.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.609+0000 7f6674d72640 10 monclient: build_initial_monmap 2024-03-25T18:19:54.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.609+0000 7f6674d72640 10 monclient: monmap: 2024-03-25T18:19:54.613 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:19:54.613 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:54.613 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:19:54.613 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:19:54.613 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:54.613 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:19:54.613 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:19:54.613 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:19:54.613 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:19:54.613 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:54.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.610+0000 7f6674d72640 1 librados: starting msgr at 2024-03-25T18:19:54.613 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.610+0000 7f6674d72640 1 librados: starting objecter 2024-03-25T18:19:54.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.610+0000 7f6674d72640 1 Processor -- start 2024-03-25T18:19:54.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.610+0000 7f6674d72640 1 -- start start 2024-03-25T18:19:54.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.610+0000 7f6674d72640 1 librados: setting wanted keys 2024-03-25T18:19:54.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.610+0000 7f6674d72640 1 librados: calling monclient init 2024-03-25T18:19:54.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.610+0000 7f6674d72640 10 monclient: init 2024-03-25T18:19:54.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.610+0000 7f6674d72640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:54.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.610+0000 7f6674d72640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:54.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.610+0000 7f6674d72640 10 monclient(hunting): picked mon.noname-b con 0x7f66700a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:19:54.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.610+0000 7f6674d72640 10 monclient(hunting): picked mon.noname-a con 0x7f667013f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:19:54.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.610+0000 7f6674d72640 10 monclient(hunting): picked mon.noname-c con 0x7f66701430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:19:54.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.610+0000 7f6674d72640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6670139a10 con 0x7f667013f710 2024-03-25T18:19:54.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.610+0000 7f6674d72640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66701d7880 con 0x7f66700a98f0 2024-03-25T18:19:54.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.610+0000 7f6674d72640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6670137890 con 0x7f66701430c0 2024-03-25T18:19:54.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.610+0000 7f6674d72640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:54.614 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.610+0000 7f6674d72640 10 monclient(hunting): authenticate will time out at 7580.849408s 2024-03-25T18:19:54.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.618+0000 7f666e59c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f66700a98f0 0x7f66701d7da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:58504/0 (socket says 172.21.15.64:58504) 2024-03-25T18:19:54.619 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.618+0000 7f666e59c640 1 -- 172.21.15.64:0/1412037192 learned_addr learned my addr 172.21.15.64:0/1412037192 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:19:54.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.619+0000 7f665effd640 10 client.?.objecter ms_handle_connect 0x7f66701430c0 2024-03-25T18:19:54.620 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.619+0000 7f665effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:54.623 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.623+0000 7f665effd640 10 client.?.objecter ms_handle_connect 0x7f66700a98f0 2024-03-25T18:19:54.624 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.623+0000 7f665effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:54.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.629+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802472352 0 0) 0x7f6670137890 con 0x7f66701430c0 2024-03-25T18:19:54.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.629+0000 7f665effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.629+0000 7f665effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.629+0000 7f665effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.629+0000 7f665effd640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:19:54.630 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.629+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6650004eb0 con 0x7f66701430c0 2024-03-25T18:19:54.635 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.634+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528624681 0 0) 0x7f66701d7880 con 0x7f66700a98f0 2024-03-25T18:19:54.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.634+0000 7f665effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.634+0000 7f665effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.634+0000 7f665effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.634+0000 7f665effd640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:19:54.636 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.634+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6670137890 con 0x7f66700a98f0 2024-03-25T18:19:54.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.638+0000 7f665effd640 10 client.?.objecter ms_handle_connect 0x7f667013f710 2024-03-25T18:19:54.639 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.638+0000 7f665effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:54.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.652+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798768270 0 0) 0x7f6670139a10 con 0x7f667013f710 2024-03-25T18:19:54.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.652+0000 7f665effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.652+0000 7f665effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.653 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.652+0000 7f665effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.652+0000 7f665effd640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:19:54.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.652+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66701d7880 con 0x7f667013f710 2024-03-25T18:19:54.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.658+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2881958620 0 0) 0x7f6670137890 con 0x7f66700a98f0 2024-03-25T18:19:54.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.658+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6670139a10 con 0x7f66700a98f0 2024-03-25T18:19:54.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.658+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f6660001930 con 0x7f66700a98f0 2024-03-25T18:19:54.659 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.658+0000 7f665effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.658+0000 7f665effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:54.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.658+0000 7f665effd640 10 monclient(hunting): dump: 2024-03-25T18:19:54.660 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:54.660 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:54.660 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:54.660 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:54.660 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.660 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:54.660 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:54.660 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:54.660 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:54.660 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:54.660 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.658+0000 7f665effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:54.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.674+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3260594184 0 0) 0x7f66701d7880 con 0x7f667013f710 2024-03-25T18:19:54.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.674+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6670137890 con 0x7f667013f710 2024-03-25T18:19:54.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.674+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f66640017f0 con 0x7f667013f710 2024-03-25T18:19:54.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.674+0000 7f665effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.674+0000 7f665effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:54.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.674+0000 7f665effd640 10 monclient(hunting): dump: 2024-03-25T18:19:54.676 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:54.676 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:54.676 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:54.676 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:54.676 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.676 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:54.676 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:54.676 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:54.676 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:54.676 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:54.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.678+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3719539111 0 0) 0x7f6650004eb0 con 0x7f66701430c0 2024-03-25T18:19:54.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.679+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66701d7880 con 0x7f66701430c0 2024-03-25T18:19:54.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.679+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f6668001e80 con 0x7f66701430c0 2024-03-25T18:19:54.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.679+0000 7f665effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.679+0000 7f665effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:54.680 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.679+0000 7f665effd640 10 monclient(hunting): dump: 2024-03-25T18:19:54.680 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:54.680 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:54.680 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:54.680 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:54.680 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.681 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:54.681 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:54.681 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:54.681 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:54.681 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:54.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3213973437 0 0) 0x7f6670139a10 con 0x7f66700a98f0 2024-03-25T18:19:54.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 >> v1:172.21.15.64:6790/0 conn(0x7f66701430c0 legacy=0x7f66701e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 >> v1:172.21.15.64:6789/0 conn(0x7f667013f710 legacy=0x7f6670138370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f665effd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:54.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f665effd640 1 monclient: found mon.b 2024-03-25T18:19:54.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f665effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:54.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f665effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:19:54.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66701d84b0 con 0x7f66700a98f0 2024-03-25T18:19:54.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f665effd640 10 monclient: _finish_auth 0 2024-03-25T18:19:54.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f665effd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:54.685 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f665effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f6674d72640 5 monclient: authenticate success, global_id 4693 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f6674d72640 10 monclient: _renew_subs 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f6674d72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f6674d72640 1 -- 172.21.15.64:0/1412037192 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66701d51f0 con 0x7f66700a98f0 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f6674d72640 10 client.4693.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f6674d72640 10 monclient: _renew_subs 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f6674d72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f6674d72640 1 -- 172.21.15.64:0/1412037192 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6670138c60 con 0x7f66700a98f0 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.683+0000 7f6674d72640 1 librados: init done 2024-03-25T18:19:54.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.684+0000 7f6674d72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:54.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.684+0000 7f6674d72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:54.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.684+0000 7f6674d72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:19:54.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.684+0000 7f6674d72640 1 -- 172.21.15.64:0/1412037192 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66700627b0 con 0x7f66700a98f0 2024-03-25T18:19:54.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.959+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6660001d60 con 0x7f66700a98f0 2024-03-25T18:19:54.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.959+0000 7f665effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:54.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.959+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f6660004a40 con 0x7f66700a98f0 2024-03-25T18:19:54.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.959+0000 7f665effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.961 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.959+0000 7f665effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:54.962 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.959+0000 7f665effd640 10 monclient: dump: 2024-03-25T18:19:54.962 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:19:54.962 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:19:54.962 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:54.962 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:19:54.963 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.963 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:19:54.963 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:19:54.963 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:19:54.963 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:19:54.964 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:19:54.964 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.959+0000 7f666cd99640 4 set_mon_vals no callback set 2024-03-25T18:19:54.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.960+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f666001aad0 con 0x7f66700a98f0 2024-03-25T18:19:54.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.961+0000 7f666dd9b640 10 monclient: get_auth_request con 0x7f665006e0b0 auth_method 0 2024-03-25T18:19:54.965 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.961+0000 7f665effd640 10 client.4693.objecter ms_handle_connect 0x7f665006e0b0 2024-03-25T18:19:54.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.964+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4227240091 0 0) 0x7f66600547d0 con 0x7f66700a98f0 2024-03-25T18:19:54.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.964+0000 7f665effd640 10 client.4693.objecter ms_dispatch 0x7f6670000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:54.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.964+0000 7f665effd640 3 client.4693.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:54.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.964+0000 7f665effd640 3 client.4693.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:54.967 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.964+0000 7f665effd640 20 client.4693.objecter dump_active .. 0 homeless 2024-03-25T18:19:54.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.965+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6660050110 con 0x7f66700a98f0 2024-03-25T18:19:54.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.965+0000 7f665effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:54.968 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:54.965+0000 7f665effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:55.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.235+0000 7f6674d72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:55.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.235+0000 7f6674d72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:55.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.235+0000 7f6674d72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:19:55.236 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.235+0000 7f6674d72640 1 -- 172.21.15.64:0/1412037192 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f667013df90 con 0x7f66700a98f0 2024-03-25T18:19:55.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.264+0000 7f665effd640 1 -- 172.21.15.64:0/1412037192 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 991200065) 0x7f6660052210 con 0x7f66700a98f0 2024-03-25T18:19:55.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.264+0000 7f665effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:55.266 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.264+0000 7f665effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:55.266 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:19:55.266 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:19:38.421162+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:38.421162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:55.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.267+0000 7f6674d72640 10 librados: watch_flush enter 2024-03-25T18:19:55.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.267+0000 7f6674d72640 10 librados: watch_flush exit 2024-03-25T18:19:55.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.267+0000 7f6674d72640 20 client.4693.objecter shutdown clearing up homeless session... 2024-03-25T18:19:55.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.267+0000 7f6674d72640 10 client.4693.objecter successfully canceled tick 2024-03-25T18:19:55.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.267+0000 7f6674d72640 1 -- 172.21.15.64:0/1412037192 >> v1:172.21.15.64:6800/2915543638 conn(0x7f665006e0b0 legacy=0x7f6650070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:55.269 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.267+0000 7f6674d72640 10 monclient: shutdown 2024-03-25T18:19:55.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.267+0000 7f6674d72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:55.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.267+0000 7f6674d72640 1 -- 172.21.15.64:0/1412037192 >> v1:172.21.15.86:6789/0 conn(0x7f66700a98f0 legacy=0x7f66701d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:55.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.267+0000 7f6674d72640 1 -- 172.21.15.64:0/1412037192 shutdown_connections 2024-03-25T18:19:55.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.267+0000 7f6674d72640 1 -- 172.21.15.64:0/1412037192 >> 172.21.15.64:0/1412037192 conn(0x7f66700b7ea0 msgr2=0x7f6670074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:55.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.267+0000 7f6674d72640 1 -- 172.21.15.64:0/1412037192 shutdown_connections 2024-03-25T18:19:55.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.267+0000 7f6674d72640 1 -- 172.21.15.64:0/1412037192 wait complete. 2024-03-25T18:19:55.270 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:19:55.268+0000 7f6674d72640 1 librados: shutdown 2024-03-25T18:19:55.281 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:02.086 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:02.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.202+0000 7fbe14033640 10 monclient: get_monmap_and_config 2024-03-25T18:20:02.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.202+0000 7fbe14033640 10 monclient: build_initial_monmap 2024-03-25T18:20:02.203 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.202+0000 7fbe14033640 10 monclient: monmap: 2024-03-25T18:20:02.204 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:20:02.204 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:02.204 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:20:02.203766+0000 2024-03-25T18:20:02.204 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:20:02.203766+0000 2024-03-25T18:20:02.204 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:02.204 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:02.204 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:20:02.204 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:20:02.204 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:20:02.204 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:02.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.203+0000 7fbe14033640 1 Processor -- start 2024-03-25T18:20:02.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.203+0000 7fbe14033640 1 -- start start 2024-03-25T18:20:02.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.203+0000 7fbe14033640 10 monclient: init 2024-03-25T18:20:02.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.204+0000 7fbe14033640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:02.205 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.204+0000 7fbe14033640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:02.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.204+0000 7fbe14033640 10 monclient(hunting): picked mon.noname-c con 0x7fbe0c0a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:20:02.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.204+0000 7fbe14033640 10 monclient(hunting): picked mon.noname-b con 0x7fbe0c13f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:20:02.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.204+0000 7fbe14033640 10 monclient(hunting): picked mon.noname-a con 0x7fbe0c1430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:20:02.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.204+0000 7fbe14033640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe0c142520 con 0x7fbe0c1430c0 2024-03-25T18:20:02.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.204+0000 7fbe14033640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe0c0b28a0 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.204+0000 7fbe14033640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe0c0b2a30 con 0x7fbe0c13f710 2024-03-25T18:20:02.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.204+0000 7fbe14033640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:02.206 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.204+0000 7fbe14033640 10 monclient(hunting): authenticate will time out at 7588.442843s 2024-03-25T18:20:02.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.213+0000 7fbe115ce640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fbe0c13f710 0x7fbe0c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:45606/0 (socket says 172.21.15.64:45606) 2024-03-25T18:20:02.214 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.213+0000 7fbe115ce640 1 -- 172.21.15.64:0/3931614613 learned_addr learned my addr 172.21.15.64:0/3931614613 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:20:02.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.219+0000 7fbe10dcd640 1 -- 172.21.15.64:0/3931614613 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2072088037 0 0) 0x7fbe0c0b2a30 con 0x7fbe0c13f710 2024-03-25T18:20:02.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.219+0000 7fbe10dcd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.219+0000 7fbe10dcd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.219+0000 7fbe10dcd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.219+0000 7fbe10dcd640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:20:02.220 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.219+0000 7fbe10dcd640 1 -- 172.21.15.64:0/3931614613 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf8003520 con 0x7fbe0c13f710 2024-03-25T18:20:02.228 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.227+0000 7fbe10dcd640 1 -- 172.21.15.64:0/3931614613 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3976013773 0 0) 0x7fbdf8003520 con 0x7fbe0c13f710 2024-03-25T18:20:02.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.227+0000 7fbe10dcd640 1 -- 172.21.15.64:0/3931614613 >> v1:172.21.15.86:6789/0 conn(0x7fbe0c0a98f0 legacy=0x7fbe0c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.227+0000 7fbe10dcd640 1 -- 172.21.15.64:0/3931614613 >> v1:172.21.15.64:6789/0 conn(0x7fbe0c1430c0 legacy=0x7fbe0c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.227+0000 7fbe10dcd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:02.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.227+0000 7fbe10dcd640 1 monclient: found mon.noname-b 2024-03-25T18:20:02.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.227+0000 7fbe10dcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:02.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.227+0000 7fbe10dcd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:20:02.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.227+0000 7fbe10dcd640 1 -- 172.21.15.64:0/3931614613 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0c0b44c0 con 0x7fbe0c13f710 2024-03-25T18:20:02.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.227+0000 7fbe10dcd640 10 monclient: _finish_auth 0 2024-03-25T18:20:02.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.227+0000 7fbe10dcd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:02.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.227+0000 7fbe10dcd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:02.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.227+0000 7fbe10dcd640 1 -- 172.21.15.64:0/3931614613 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbdfc001690 con 0x7fbe0c13f710 2024-03-25T18:20:02.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.227+0000 7fbe10dcd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.227+0000 7fbe10dcd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:02.229 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.227+0000 7fbe10dcd640 10 monclient: dump: 2024-03-25T18:20:02.229 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:02.229 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:02.230 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:02.230 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:02.230 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.230 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:02.230 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:02.230 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:02.230 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:02.230 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:02.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.227+0000 7fbe14033640 5 monclient: authenticate success, global_id 4691 2024-03-25T18:20:02.230 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.227+0000 7fbe14033640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:02.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.237+0000 7fbe10dcd640 1 -- 172.21.15.64:0/3931614613 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdfc0017f0 con 0x7fbe0c13f710 2024-03-25T18:20:02.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.237+0000 7fbe10dcd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:02.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.237+0000 7fbe10dcd640 1 -- 172.21.15.64:0/3931614613 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbdfc001c00 con 0x7fbe0c13f710 2024-03-25T18:20:02.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.237+0000 7fbe14033640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:02.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.237+0000 7fbe14033640 4 set_mon_vals no callback set 2024-03-25T18:20:02.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.237+0000 7fbe10dcd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.237+0000 7fbe10dcd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:02.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.237+0000 7fbe10dcd640 10 monclient: dump: 2024-03-25T18:20:02.240 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:02.240 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:02.240 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:02.240 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:02.240 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.240 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:02.240 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:02.240 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:02.240 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:02.240 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:02.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.238+0000 7fbe14033640 10 monclient: shutdown 2024-03-25T18:20:02.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.238+0000 7fbe14033640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:02.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.238+0000 7fbe14033640 1 -- 172.21.15.64:0/3931614613 >> v1:172.21.15.64:6790/0 conn(0x7fbe0c13f710 legacy=0x7fbe0c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.238+0000 7fbe14033640 1 -- 172.21.15.64:0/3931614613 shutdown_connections 2024-03-25T18:20:02.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.238+0000 7fbe14033640 1 -- 172.21.15.64:0/3931614613 >> 172.21.15.64:0/3931614613 conn(0x7fbe0c0b7ea0 msgr2=0x7fbe0c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:02.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.238+0000 7fbe14033640 1 -- 172.21.15.64:0/3931614613 shutdown_connections 2024-03-25T18:20:02.241 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.238+0000 7fbe14033640 1 -- 172.21.15.64:0/3931614613 wait complete. 2024-03-25T18:20:02.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.239+0000 7fbe14033640 10 monclient: build_initial_monmap 2024-03-25T18:20:02.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.239+0000 7fbe14033640 10 monclient: monmap: 2024-03-25T18:20:02.243 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:20:02.243 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:02.243 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:20:02.243 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:20:02.243 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:02.243 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:20:02.243 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:20:02.243 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:20:02.243 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:20:02.243 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:02.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.239+0000 7fbe14033640 1 librados: starting msgr at 2024-03-25T18:20:02.243 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.239+0000 7fbe14033640 1 librados: starting objecter 2024-03-25T18:20:02.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.240+0000 7fbe14033640 1 Processor -- start 2024-03-25T18:20:02.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.240+0000 7fbe14033640 1 -- start start 2024-03-25T18:20:02.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.240+0000 7fbe14033640 1 librados: setting wanted keys 2024-03-25T18:20:02.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.240+0000 7fbe14033640 1 librados: calling monclient init 2024-03-25T18:20:02.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.240+0000 7fbe14033640 10 monclient: init 2024-03-25T18:20:02.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.241+0000 7fbe14033640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:02.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.241+0000 7fbe14033640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:02.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.241+0000 7fbe14033640 10 monclient(hunting): picked mon.noname-b con 0x7fbe0c0a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:20:02.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.241+0000 7fbe14033640 10 monclient(hunting): picked mon.noname-a con 0x7fbe0c1430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:20:02.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.241+0000 7fbe14033640 10 monclient(hunting): picked mon.noname-c con 0x7fbe0c136ab0 addr v1:172.21.15.64:6790/0 2024-03-25T18:20:02.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.241+0000 7fbe14033640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe0c0ac0c0 con 0x7fbe0c1430c0 2024-03-25T18:20:02.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.241+0000 7fbe14033640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe0c0ac3a0 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.241+0000 7fbe14033640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe0c1380c0 con 0x7fbe0c136ab0 2024-03-25T18:20:02.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.241+0000 7fbe14033640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:02.244 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.241+0000 7fbe14033640 10 monclient(hunting): authenticate will time out at 7588.479875s 2024-03-25T18:20:02.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.255+0000 7fbe115ce640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7fbe0c1430c0 0x7fbe0c138e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:51602/0 (socket says 172.21.15.64:51602) 2024-03-25T18:20:02.256 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.255+0000 7fbe115ce640 1 -- 172.21.15.64:0/2390855008 learned_addr learned my addr 172.21.15.64:0/2390855008 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:20:02.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.259+0000 7fbe027fc640 10 client.?.objecter ms_handle_connect 0x7fbe0c1430c0 2024-03-25T18:20:02.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.259+0000 7fbe027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:02.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.259+0000 7fbe027fc640 10 client.?.objecter ms_handle_connect 0x7fbe0c0a98f0 2024-03-25T18:20:02.260 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.259+0000 7fbe027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:02.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.267+0000 7fbe027fc640 10 client.?.objecter ms_handle_connect 0x7fbe0c136ab0 2024-03-25T18:20:02.268 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.267+0000 7fbe027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:02.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.272+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877871304 0 0) 0x7fbe0c0ac3a0 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.272+0000 7fbe027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.272+0000 7fbe027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.273 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.272+0000 7fbe027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.272+0000 7fbe027fc640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:20:02.274 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.272+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf0004eb0 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.277+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820489812 0 0) 0x7fbe0c0ac0c0 con 0x7fbe0c1430c0 2024-03-25T18:20:02.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.277+0000 7fbe027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.277+0000 7fbe027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.277+0000 7fbe027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.277+0000 7fbe027fc640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:20:02.278 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.277+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe0c0ac3a0 con 0x7fbe0c1430c0 2024-03-25T18:20:02.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.281+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786747211 0 0) 0x7fbe0c1380c0 con 0x7fbe0c136ab0 2024-03-25T18:20:02.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.281+0000 7fbe027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.281+0000 7fbe027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.281+0000 7fbe027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.281+0000 7fbe027fc640 10 monclient(hunting): my global_id is 4694 2024-03-25T18:20:02.282 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.281+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe0c0ac0c0 con 0x7fbe0c136ab0 2024-03-25T18:20:02.290 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.289+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3407769706 0 0) 0x7fbe0c0ac0c0 con 0x7fbe0c136ab0 2024-03-25T18:20:02.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.289+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe0c1380c0 con 0x7fbe0c136ab0 2024-03-25T18:20:02.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.289+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 305929961 0 0) 0x7fbdf0004eb0 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.289+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe0c0ac0c0 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.289+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbe04001a90 con 0x7fbe0c136ab0 2024-03-25T18:20:02.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.289+0000 7fbe027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.289+0000 7fbe027fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:02.291 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.289+0000 7fbe027fc640 10 monclient(hunting): dump: 2024-03-25T18:20:02.291 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:02.291 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:02.291 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:02.291 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:02.291 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.292 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:02.292 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:02.292 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:02.292 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:02.292 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:02.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.289+0000 7fbe027fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:02.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.289+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbe08001840 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.289+0000 7fbe027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.289+0000 7fbe027fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:02.292 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.289+0000 7fbe027fc640 10 monclient(hunting): dump: 2024-03-25T18:20:02.292 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:02.292 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:02.292 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:02.292 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:02.292 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.292 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:02.293 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:02.293 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:02.293 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:02.293 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:02.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.296+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 382906148 0 0) 0x7fbe0c0ac3a0 con 0x7fbe0c1430c0 2024-03-25T18:20:02.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.296+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdf0004eb0 con 0x7fbe0c1430c0 2024-03-25T18:20:02.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.296+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbdfc001b00 con 0x7fbe0c1430c0 2024-03-25T18:20:02.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.296+0000 7fbe027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.296+0000 7fbe027fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:02.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.296+0000 7fbe027fc640 10 monclient(hunting): dump: 2024-03-25T18:20:02.297 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:02.297 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:02.297 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:02.297 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:02.297 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.297 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:02.297 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:02.298 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:02.298 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:02.298 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:02.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.296+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 609570190 0 0) 0x7fbe0c0ac0c0 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 >> v1:172.21.15.64:6790/0 conn(0x7fbe0c136ab0 legacy=0x7fbe0c1e2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 >> v1:172.21.15.64:6789/0 conn(0x7fbe0c1430c0 legacy=0x7fbe0c138e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe027fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:02.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe027fc640 1 monclient: found mon.b 2024-03-25T18:20:02.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:02.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe027fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:02.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0c0acf40 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe027fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:02.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe027fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:02.299 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe14033640 5 monclient: authenticate success, global_id 4699 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe14033640 10 monclient: _renew_subs 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe14033640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe14033640 1 -- 172.21.15.64:0/2390855008 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe0c1395a0 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe14033640 10 client.4699.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe14033640 10 monclient: _renew_subs 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe14033640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe14033640 1 -- 172.21.15.64:0/2390855008 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0c1e4020 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.297+0000 7fbe14033640 1 librados: init done 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.298+0000 7fbe14033640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.298+0000 7fbe14033640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.298+0000 7fbe14033640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.298+0000 7fbe14033640 1 -- 172.21.15.64:0/2390855008 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe0c138250 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.586+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe080019a0 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.586+0000 7fbe027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:02.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.586+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fbe080035f0 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.586+0000 7fbe027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.586+0000 7fbe027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:02.587 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.586+0000 7fbe027fc640 10 monclient: dump: 2024-03-25T18:20:02.587 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:02.588 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:02.588 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:02.588 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:02.588 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.588 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:02.588 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:02.588 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:02.588 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:02.588 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:02.588 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.586+0000 7fbe03fff640 4 set_mon_vals no callback set 2024-03-25T18:20:02.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.587+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fbe08019680 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.587+0000 7fbe115ce640 10 monclient: get_auth_request con 0x7fbdf006e010 auth_method 0 2024-03-25T18:20:02.589 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.587+0000 7fbe027fc640 10 client.4699.objecter ms_handle_connect 0x7fbdf006e010 2024-03-25T18:20:02.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.592+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4227240091 0 0) 0x7fbe08053420 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.592+0000 7fbe027fc640 10 client.4699.objecter ms_dispatch 0x7fbe0c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:20:02.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.592+0000 7fbe027fc640 3 client.4699.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:20:02.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.592+0000 7fbe027fc640 3 client.4699.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:20:02.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.592+0000 7fbe027fc640 20 client.4699.objecter dump_active .. 0 homeless 2024-03-25T18:20:02.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.592+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe08052af0 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.592+0000 7fbe027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:02.593 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.592+0000 7fbe027fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:02.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.845+0000 7fbe14033640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:02.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.845+0000 7fbe14033640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:02.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.845+0000 7fbe14033640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:02.846 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.845+0000 7fbe14033640 1 -- 172.21.15.64:0/2390855008 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe0c0627b0 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.876+0000 7fbe027fc640 1 -- 172.21.15.64:0/2390855008 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 991200065) 0x7fbe08050e60 con 0x7fbe0c0a98f0 2024-03-25T18:20:02.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.876+0000 7fbe027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:02.877 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.876+0000 7fbe027fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:02.877 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:20:02.878 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:19:38.421162+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:38.421162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:02.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.878+0000 7fbe14033640 10 librados: watch_flush enter 2024-03-25T18:20:02.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.878+0000 7fbe14033640 10 librados: watch_flush exit 2024-03-25T18:20:02.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.878+0000 7fbe14033640 20 client.4699.objecter shutdown clearing up homeless session... 2024-03-25T18:20:02.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.878+0000 7fbe14033640 10 client.4699.objecter successfully canceled tick 2024-03-25T18:20:02.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.878+0000 7fbe14033640 1 -- 172.21.15.64:0/2390855008 >> v1:172.21.15.64:6800/2915543638 conn(0x7fbdf006e010 legacy=0x7fbdf00704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.878+0000 7fbe14033640 10 monclient: shutdown 2024-03-25T18:20:02.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.878+0000 7fbe14033640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:02.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.878+0000 7fbe14033640 1 -- 172.21.15.64:0/2390855008 >> v1:172.21.15.86:6789/0 conn(0x7fbe0c0a98f0 legacy=0x7fbe0c0ac830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.878+0000 7fbe14033640 1 -- 172.21.15.64:0/2390855008 shutdown_connections 2024-03-25T18:20:02.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.878+0000 7fbe14033640 1 -- 172.21.15.64:0/2390855008 >> 172.21.15.64:0/2390855008 conn(0x7fbe0c0b7ea0 msgr2=0x7fbe0c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:02.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.878+0000 7fbe14033640 1 -- 172.21.15.64:0/2390855008 shutdown_connections 2024-03-25T18:20:02.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.879+0000 7fbe14033640 1 -- 172.21.15.64:0/2390855008 wait complete. 2024-03-25T18:20:02.880 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:02.879+0000 7fbe14033640 1 librados: shutdown 2024-03-25T18:20:02.891 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:03.549 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:10:46.661 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:03.602 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:10:46.715 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:03.622 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:10:46.735 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:03.649 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:10:46.728 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:03.655 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:10:46.734 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:03.662 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:10:46.775 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:03.668 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:10:46.780 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:03.698 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:10:46.776 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:03.706 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:10:46.818 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:09.698 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:09.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.813+0000 7f83133b6640 10 monclient: get_monmap_and_config 2024-03-25T18:20:09.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.813+0000 7f83133b6640 10 monclient: build_initial_monmap 2024-03-25T18:20:09.814 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.813+0000 7f83133b6640 10 monclient: monmap: 2024-03-25T18:20:09.814 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:20:09.815 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:09.815 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:20:09.814337+0000 2024-03-25T18:20:09.815 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:20:09.814337+0000 2024-03-25T18:20:09.815 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:09.815 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:09.815 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:20:09.815 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:20:09.815 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:20:09.815 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.814+0000 7f83133b6640 1 Processor -- start 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.814+0000 7f83133b6640 1 -- start start 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.814+0000 7f83133b6640 10 monclient: init 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.815+0000 7f83133b6640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.815+0000 7f83133b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.815+0000 7f83133b6640 10 monclient(hunting): picked mon.noname-b con 0x7f830c0a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.815+0000 7f83133b6640 10 monclient(hunting): picked mon.noname-c con 0x7f830c13f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.815+0000 7f83133b6640 10 monclient(hunting): picked mon.noname-a con 0x7f830c1430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:20:09.816 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.815+0000 7f83133b6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830c142520 con 0x7f830c1430c0 2024-03-25T18:20:09.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.815+0000 7f83133b6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830c0b28a0 con 0x7f830c13f710 2024-03-25T18:20:09.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.815+0000 7f83133b6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830c0b2a30 con 0x7f830c0a98f0 2024-03-25T18:20:09.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.815+0000 7f83133b6640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:09.817 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.815+0000 7f83133b6640 10 monclient(hunting): authenticate will time out at 7596.053746s 2024-03-25T18:20:09.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.825+0000 7f8311953640 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f830c1430c0 0x7f830c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:38358/0 (socket says 172.21.15.64:38358) 2024-03-25T18:20:09.826 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.825+0000 7f8311953640 1 -- 172.21.15.64:0/1842664520 learned_addr learned my addr 172.21.15.64:0/1842664520 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:20:09.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.837+0000 7f8303fff640 1 -- 172.21.15.64:0/1842664520 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817145205 0 0) 0x7f830c0b2a30 con 0x7f830c0a98f0 2024-03-25T18:20:09.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.837+0000 7f8303fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.837+0000 7f8303fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.837+0000 7f8303fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.837+0000 7f8303fff640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:20:09.838 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.837+0000 7f8303fff640 1 -- 172.21.15.64:0/1842664520 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82f8003520 con 0x7f830c0a98f0 2024-03-25T18:20:09.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.839+0000 7f8303fff640 1 -- 172.21.15.64:0/1842664520 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754046521 0 0) 0x7f830c142520 con 0x7f830c1430c0 2024-03-25T18:20:09.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.839+0000 7f8303fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.839+0000 7f8303fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.839+0000 7f8303fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.839+0000 7f8303fff640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:20:09.840 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.839+0000 7f8303fff640 1 -- 172.21.15.64:0/1842664520 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f830c0b2a30 con 0x7f830c1430c0 2024-03-25T18:20:09.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.848+0000 7f8303fff640 1 -- 172.21.15.64:0/1842664520 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213906058 0 0) 0x7f830c0b28a0 con 0x7f830c13f710 2024-03-25T18:20:09.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.848+0000 7f8303fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.848+0000 7f8303fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.848+0000 7f8303fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.848+0000 7f8303fff640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:20:09.849 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.848+0000 7f8303fff640 1 -- 172.21.15.64:0/1842664520 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f830c142520 con 0x7f830c13f710 2024-03-25T18:20:09.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.851+0000 7f8303fff640 1 -- 172.21.15.64:0/1842664520 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1822033798 0 0) 0x7f82f8003520 con 0x7f830c0a98f0 2024-03-25T18:20:09.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.851+0000 7f8303fff640 1 -- 172.21.15.64:0/1842664520 >> v1:172.21.15.86:6789/0 conn(0x7f830c13f710 legacy=0x7f830c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.851+0000 7f8303fff640 1 -- 172.21.15.64:0/1842664520 >> v1:172.21.15.64:6789/0 conn(0x7f830c1430c0 legacy=0x7f830c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.851+0000 7f8303fff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:09.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.851+0000 7f8303fff640 1 monclient: found mon.noname-b 2024-03-25T18:20:09.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.851+0000 7f8303fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:09.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.851+0000 7f8303fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:20:09.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.851+0000 7f8303fff640 1 -- 172.21.15.64:0/1842664520 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830c0b44c0 con 0x7f830c0a98f0 2024-03-25T18:20:09.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.851+0000 7f8303fff640 10 monclient: _finish_auth 0 2024-03-25T18:20:09.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.851+0000 7f8303fff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:09.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.851+0000 7f8303fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:09.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.851+0000 7f8303fff640 1 -- 172.21.15.64:0/1842664520 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f8308001690 con 0x7f830c0a98f0 2024-03-25T18:20:09.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.851+0000 7f8303fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:09.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.851+0000 7f8303fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:09.853 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.851+0000 7f8303fff640 10 monclient: dump: 2024-03-25T18:20:09.853 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:09.854 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:09.854 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:09.854 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:09.854 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:09.854 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:09.854 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:09.854 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:09.854 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:09.854 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:09.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.851+0000 7f83133b6640 5 monclient: authenticate success, global_id 4697 2024-03-25T18:20:09.854 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.851+0000 7f83133b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:09.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.857+0000 7f8303fff640 1 -- 172.21.15.64:0/1842664520 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83080017f0 con 0x7f830c0a98f0 2024-03-25T18:20:09.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.857+0000 7f8303fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:09.858 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.857+0000 7f8303fff640 1 -- 172.21.15.64:0/1842664520 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f8308001c00 con 0x7f830c0a98f0 2024-03-25T18:20:09.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.857+0000 7f8303fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:09.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.857+0000 7f8303fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:09.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.857+0000 7f8303fff640 10 monclient: dump: 2024-03-25T18:20:09.859 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:09.859 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:09.859 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:09.859 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:09.859 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:09.859 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:09.859 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:09.859 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:09.859 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:09.859 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:09.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.857+0000 7f83133b6640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:09.859 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.857+0000 7f83133b6640 4 set_mon_vals no callback set 2024-03-25T18:20:09.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.858+0000 7f83133b6640 10 monclient: shutdown 2024-03-25T18:20:09.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.858+0000 7f83133b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:09.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.858+0000 7f83133b6640 1 -- 172.21.15.64:0/1842664520 >> v1:172.21.15.64:6790/0 conn(0x7f830c0a98f0 legacy=0x7f830c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.858+0000 7f83133b6640 1 -- 172.21.15.64:0/1842664520 shutdown_connections 2024-03-25T18:20:09.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.858+0000 7f83133b6640 1 -- 172.21.15.64:0/1842664520 >> 172.21.15.64:0/1842664520 conn(0x7f830c0b7ea0 msgr2=0x7f830c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:09.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.858+0000 7f83133b6640 1 -- 172.21.15.64:0/1842664520 shutdown_connections 2024-03-25T18:20:09.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.858+0000 7f83133b6640 1 -- 172.21.15.64:0/1842664520 wait complete. 2024-03-25T18:20:09.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.859+0000 7f83133b6640 10 monclient: build_initial_monmap 2024-03-25T18:20:09.861 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.859+0000 7f83133b6640 10 monclient: monmap: 2024-03-25T18:20:09.861 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:20:09.861 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:09.861 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:20:09.861 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:20:09.861 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:09.861 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:20:09.862 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:20:09.862 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:20:09.862 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:20:09.862 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:09.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.859+0000 7f83133b6640 1 librados: starting msgr at 2024-03-25T18:20:09.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.859+0000 7f83133b6640 1 librados: starting objecter 2024-03-25T18:20:09.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.859+0000 7f83133b6640 1 Processor -- start 2024-03-25T18:20:09.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.859+0000 7f83133b6640 1 -- start start 2024-03-25T18:20:09.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.859+0000 7f83133b6640 1 librados: setting wanted keys 2024-03-25T18:20:09.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.859+0000 7f83133b6640 1 librados: calling monclient init 2024-03-25T18:20:09.862 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.859+0000 7f83133b6640 10 monclient: init 2024-03-25T18:20:09.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.860+0000 7f83133b6640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:09.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.860+0000 7f83133b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:09.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.860+0000 7f83133b6640 10 monclient(hunting): picked mon.noname-b con 0x7f830c0a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:20:09.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.860+0000 7f83133b6640 10 monclient(hunting): picked mon.noname-c con 0x7f830c13f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:20:09.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.860+0000 7f83133b6640 10 monclient(hunting): picked mon.noname-a con 0x7f830c1430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:20:09.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.860+0000 7f83133b6640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830c138b20 con 0x7f830c1430c0 2024-03-25T18:20:09.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.860+0000 7f83133b6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830c137a80 con 0x7f830c0a98f0 2024-03-25T18:20:09.863 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.860+0000 7f83133b6640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830c139880 con 0x7f830c13f710 2024-03-25T18:20:09.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.860+0000 7f83133b6640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:09.864 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.860+0000 7f83133b6640 10 monclient(hunting): authenticate will time out at 7596.099278s 2024-03-25T18:20:09.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.867+0000 7f8310951640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f830c13f710 0x7f830c139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:39978/0 (socket says 172.21.15.64:39978) 2024-03-25T18:20:09.868 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.867+0000 7f8310951640 1 -- 172.21.15.64:0/1237704194 learned_addr learned my addr 172.21.15.64:0/1237704194 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:20:09.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.872+0000 7f8301ffb640 10 client.?.objecter ms_handle_connect 0x7f830c13f710 2024-03-25T18:20:09.873 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.872+0000 7f8301ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:09.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.882+0000 7f8301ffb640 10 client.?.objecter ms_handle_connect 0x7f830c0a98f0 2024-03-25T18:20:09.883 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.882+0000 7f8301ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:09.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.887+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769857596 0 0) 0x7f830c139880 con 0x7f830c13f710 2024-03-25T18:20:09.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.887+0000 7f8301ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.887+0000 7f8301ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.887+0000 7f8301ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.887+0000 7f8301ffb640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:20:09.888 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.887+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82f0004eb0 con 0x7f830c13f710 2024-03-25T18:20:09.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.888+0000 7f8301ffb640 10 client.?.objecter ms_handle_connect 0x7f830c1430c0 2024-03-25T18:20:09.889 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.888+0000 7f8301ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:09.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.906+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117411931 0 0) 0x7f830c137a80 con 0x7f830c0a98f0 2024-03-25T18:20:09.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.906+0000 7f8301ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.906+0000 7f8301ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.906+0000 7f8301ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.906+0000 7f8301ffb640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:20:09.908 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.906+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f830c139880 con 0x7f830c0a98f0 2024-03-25T18:20:09.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.912+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885512533 0 0) 0x7f830c138b20 con 0x7f830c1430c0 2024-03-25T18:20:09.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.912+0000 7f8301ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.912+0000 7f8301ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.912+0000 7f8301ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.912+0000 7f8301ffb640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:20:09.914 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.912+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f830c137a80 con 0x7f830c1430c0 2024-03-25T18:20:09.922 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.920+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 172641119 0 0) 0x7f82f0004eb0 con 0x7f830c13f710 2024-03-25T18:20:09.922 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.920+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f830c138b20 con 0x7f830c13f710 2024-03-25T18:20:09.922 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.920+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f82fc001aa0 con 0x7f830c13f710 2024-03-25T18:20:09.922 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.920+0000 7f8301ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:09.922 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.920+0000 7f8301ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:09.922 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.920+0000 7f8301ffb640 10 monclient(hunting): dump: 2024-03-25T18:20:09.922 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:09.922 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:09.922 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:09.922 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:09.922 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:09.922 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:09.923 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:09.923 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:09.923 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:09.923 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:09.923 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.920+0000 7f8301ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:09.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.928+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2678959343 0 0) 0x7f830c139880 con 0x7f830c0a98f0 2024-03-25T18:20:09.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.928+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82f0004eb0 con 0x7f830c0a98f0 2024-03-25T18:20:09.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.928+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f8308001b00 con 0x7f830c0a98f0 2024-03-25T18:20:09.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.928+0000 7f8301ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:09.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.928+0000 7f8301ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:09.930 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.928+0000 7f8301ffb640 10 monclient(hunting): dump: 2024-03-25T18:20:09.930 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:09.931 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:09.931 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:09.931 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:09.931 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:09.931 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:09.931 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:09.931 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:09.931 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:09.931 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:09.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.945+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3472248573 0 0) 0x7f830c137a80 con 0x7f830c1430c0 2024-03-25T18:20:09.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.945+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f830c139880 con 0x7f830c1430c0 2024-03-25T18:20:09.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.946+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f8304001a30 con 0x7f830c1430c0 2024-03-25T18:20:09.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.946+0000 7f8301ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:09.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.946+0000 7f8301ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:09.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.946+0000 7f8301ffb640 10 monclient(hunting): dump: 2024-03-25T18:20:09.948 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:09.948 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:09.948 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:09.948 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:09.948 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:09.948 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:09.948 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:09.948 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:09.949 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:09.949 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:09.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.954+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1505600357 0 0) 0x7f82f0004eb0 con 0x7f830c0a98f0 2024-03-25T18:20:09.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.954+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 >> v1:172.21.15.64:6790/0 conn(0x7f830c13f710 legacy=0x7f830c139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.954+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 >> v1:172.21.15.64:6789/0 conn(0x7f830c1430c0 legacy=0x7f830c1e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.954+0000 7f8301ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:20:09.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.954+0000 7f8301ffb640 1 monclient: found mon.b 2024-03-25T18:20:09.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.954+0000 7f8301ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:09.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.954+0000 7f8301ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:09.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.954+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830c139a10 con 0x7f830c0a98f0 2024-03-25T18:20:09.957 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.954+0000 7f8301ffb640 10 monclient: _finish_auth 0 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.954+0000 7f8301ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.954+0000 7f8301ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.954+0000 7f83133b6640 5 monclient: authenticate success, global_id 4705 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.954+0000 7f83133b6640 10 monclient: _renew_subs 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.954+0000 7f83133b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.954+0000 7f83133b6640 1 -- 172.21.15.64:0/1237704194 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f830c1dab00 con 0x7f830c0a98f0 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.955+0000 7f83133b6640 10 client.4705.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.955+0000 7f83133b6640 10 monclient: _renew_subs 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.955+0000 7f83133b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.955+0000 7f83133b6640 1 -- 172.21.15.64:0/1237704194 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f830c13b310 con 0x7f830c0a98f0 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.955+0000 7f83133b6640 1 librados: init done 2024-03-25T18:20:09.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.956+0000 7f83133b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:09.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.956+0000 7f83133b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:09.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.956+0000 7f83133b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:09.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:09.956+0000 7f83133b6640 1 -- 172.21.15.64:0/1237704194 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830c074cc0 con 0x7f830c0a98f0 2024-03-25T18:20:10.238 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.236+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8308001ca0 con 0x7f830c0a98f0 2024-03-25T18:20:10.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.236+0000 7f8301ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:10.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.236+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f8308003a40 con 0x7f830c0a98f0 2024-03-25T18:20:10.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.236+0000 7f8301ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:10.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.236+0000 7f8301ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:10.239 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.236+0000 7f8301ffb640 10 monclient: dump: 2024-03-25T18:20:10.239 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:10.239 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:10.239 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:10.239 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:10.239 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:10.239 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:10.239 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:10.239 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:10.239 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:10.240 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:10.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.236+0000 7f83037fe640 4 set_mon_vals no callback set 2024-03-25T18:20:10.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.237+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f8308019a10 con 0x7f830c0a98f0 2024-03-25T18:20:10.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.238+0000 7f8310951640 10 monclient: get_auth_request con 0x7f82f006e160 auth_method 0 2024-03-25T18:20:10.240 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.238+0000 7f8301ffb640 10 client.4705.objecter ms_handle_connect 0x7f82f006e160 2024-03-25T18:20:10.245 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.243+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4227240091 0 0) 0x7f83080535c0 con 0x7f830c0a98f0 2024-03-25T18:20:10.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.243+0000 7f8301ffb640 10 client.4705.objecter ms_dispatch 0x7f830c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:20:10.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.243+0000 7f8301ffb640 3 client.4705.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:20:10.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.243+0000 7f8301ffb640 3 client.4705.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:20:10.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.243+0000 7f8301ffb640 20 client.4705.objecter dump_active .. 0 homeless 2024-03-25T18:20:10.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.244+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f830804ef00 con 0x7f830c0a98f0 2024-03-25T18:20:10.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.244+0000 7f8301ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:10.246 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.244+0000 7f8301ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:10.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.475+0000 7f83133b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:10.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.475+0000 7f83133b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:10.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.475+0000 7f83133b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:10.477 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.475+0000 7f83133b6640 1 -- 172.21.15.64:0/1237704194 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f830c0b3e80 con 0x7f830c0a98f0 2024-03-25T18:20:10.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.496+0000 7f8301ffb640 1 -- 172.21.15.64:0/1237704194 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 991200065) 0x7f8308001210 con 0x7f830c0a98f0 2024-03-25T18:20:10.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.496+0000 7f8301ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:10.498 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.496+0000 7f8301ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:10.499 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:20:10.499 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:19:38.421162+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:38.421162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:10.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.499+0000 7f83133b6640 10 librados: watch_flush enter 2024-03-25T18:20:10.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.499+0000 7f83133b6640 10 librados: watch_flush exit 2024-03-25T18:20:10.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.499+0000 7f83133b6640 20 client.4705.objecter shutdown clearing up homeless session... 2024-03-25T18:20:10.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.499+0000 7f83133b6640 10 client.4705.objecter successfully canceled tick 2024-03-25T18:20:10.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.499+0000 7f83133b6640 1 -- 172.21.15.64:0/1237704194 >> v1:172.21.15.64:6800/2915543638 conn(0x7f82f006e160 legacy=0x7f82f0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.499+0000 7f83133b6640 10 monclient: shutdown 2024-03-25T18:20:10.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.499+0000 7f83133b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:10.501 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.499+0000 7f83133b6640 1 -- 172.21.15.64:0/1237704194 >> v1:172.21.15.86:6789/0 conn(0x7f830c0a98f0 legacy=0x7f830c137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.499+0000 7f83133b6640 1 -- 172.21.15.64:0/1237704194 shutdown_connections 2024-03-25T18:20:10.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.499+0000 7f83133b6640 1 -- 172.21.15.64:0/1237704194 >> 172.21.15.64:0/1237704194 conn(0x7f830c0b7ea0 msgr2=0x7f830c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:10.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.499+0000 7f83133b6640 1 -- 172.21.15.64:0/1237704194 shutdown_connections 2024-03-25T18:20:10.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.499+0000 7f83133b6640 1 -- 172.21.15.64:0/1237704194 wait complete. 2024-03-25T18:20:10.502 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:10.499+0000 7f83133b6640 1 librados: shutdown 2024-03-25T18:20:10.513 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:17.320 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:17.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.437+0000 7f6c1856b640 10 monclient: get_monmap_and_config 2024-03-25T18:20:17.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.437+0000 7f6c1856b640 10 monclient: build_initial_monmap 2024-03-25T18:20:17.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.437+0000 7f6c1856b640 10 monclient: monmap: 2024-03-25T18:20:17.440 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:20:17.440 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:17.440 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:20:17.439783+0000 2024-03-25T18:20:17.440 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:20:17.439783+0000 2024-03-25T18:20:17.440 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:17.440 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:17.441 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:20:17.441 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:20:17.441 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:20:17.441 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:17.442 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.438+0000 7f6c1856b640 1 Processor -- start 2024-03-25T18:20:17.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.438+0000 7f6c1856b640 1 -- start start 2024-03-25T18:20:17.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.438+0000 7f6c1856b640 10 monclient: init 2024-03-25T18:20:17.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.439+0000 7f6c1856b640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:17.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.439+0000 7f6c1856b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:17.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.439+0000 7f6c1856b640 10 monclient(hunting): picked mon.noname-b con 0x7f6c100705b0 addr v1:172.21.15.64:6790/0 2024-03-25T18:20:17.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.439+0000 7f6c1856b640 10 monclient(hunting): picked mon.noname-a con 0x7f6c101402d0 addr v1:172.21.15.64:6789/0 2024-03-25T18:20:17.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.439+0000 7f6c1856b640 10 monclient(hunting): picked mon.noname-c con 0x7f6c10149ae0 addr v1:172.21.15.86:6789/0 2024-03-25T18:20:17.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.439+0000 7f6c1856b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c100718c0 con 0x7f6c101402d0 2024-03-25T18:20:17.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.440+0000 7f6c1856b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c100714c0 con 0x7f6c10149ae0 2024-03-25T18:20:17.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.440+0000 7f6c1856b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c10072b50 con 0x7f6c100705b0 2024-03-25T18:20:17.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.440+0000 7f6c1856b640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:17.444 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.440+0000 7f6c1856b640 10 monclient(hunting): authenticate will time out at 7603.679577s 2024-03-25T18:20:17.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.451+0000 7f6c16b08640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6c10149ae0 0x7f6c1014bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:35376/0 (socket says 172.21.15.64:35376) 2024-03-25T18:20:17.453 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.451+0000 7f6c16b08640 1 -- 172.21.15.64:0/3159961540 learned_addr learned my addr 172.21.15.64:0/3159961540 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:20:17.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.463+0000 7f6c15305640 1 -- 172.21.15.64:0/3159961540 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516575420 0 0) 0x7f6c100714c0 con 0x7f6c10149ae0 2024-03-25T18:20:17.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.463+0000 7f6c15305640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.463+0000 7f6c15305640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.463+0000 7f6c15305640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.463+0000 7f6c15305640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:20:17.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.463+0000 7f6c15305640 1 -- 172.21.15.64:0/3159961540 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bfc003520 con 0x7f6c10149ae0 2024-03-25T18:20:17.468 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.466+0000 7f6c15305640 1 -- 172.21.15.64:0/3159961540 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207921859 0 0) 0x7f6c10072b50 con 0x7f6c100705b0 2024-03-25T18:20:17.468 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.466+0000 7f6c15305640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.468 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.466+0000 7f6c15305640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.468 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.466+0000 7f6c15305640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.466+0000 7f6c15305640 10 monclient(hunting): my global_id is 4703 2024-03-25T18:20:17.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.466+0000 7f6c15305640 1 -- 172.21.15.64:0/3159961540 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c100714c0 con 0x7f6c100705b0 2024-03-25T18:20:17.482 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.480+0000 7f6c15305640 1 -- 172.21.15.64:0/3159961540 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3817447131 0 0) 0x7f6bfc003520 con 0x7f6c10149ae0 2024-03-25T18:20:17.482 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.480+0000 7f6c15305640 1 -- 172.21.15.64:0/3159961540 >> v1:172.21.15.64:6790/0 conn(0x7f6c100705b0 legacy=0x7f6c1013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.480+0000 7f6c15305640 1 -- 172.21.15.64:0/3159961540 >> v1:172.21.15.64:6789/0 conn(0x7f6c101402d0 legacy=0x7f6c10140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.480+0000 7f6c15305640 10 monclient: _finish_hunting 0 2024-03-25T18:20:17.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.480+0000 7f6c15305640 1 monclient: found mon.noname-c 2024-03-25T18:20:17.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.480+0000 7f6c15305640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:17.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.480+0000 7f6c15305640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:20:17.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.480+0000 7f6c15305640 1 -- 172.21.15.64:0/3159961540 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1013dd10 con 0x7f6c10149ae0 2024-03-25T18:20:17.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.480+0000 7f6c15305640 10 monclient: _finish_auth 0 2024-03-25T18:20:17.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.480+0000 7f6c15305640 10 monclient: _check_auth_tickets 2024-03-25T18:20:17.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.480+0000 7f6c15305640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:17.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.480+0000 7f6c15305640 1 -- 172.21.15.64:0/3159961540 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f6c08001a10 con 0x7f6c10149ae0 2024-03-25T18:20:17.483 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.480+0000 7f6c15305640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.480+0000 7f6c15305640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:17.484 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.480+0000 7f6c15305640 10 monclient: dump: 2024-03-25T18:20:17.484 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:17.484 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:17.484 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:17.484 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:17.484 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.484 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:17.484 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:17.484 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:17.484 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:17.484 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:17.485 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.480+0000 7f6c1856b640 5 monclient: authenticate success, global_id 4708 2024-03-25T18:20:17.485 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.480+0000 7f6c1856b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:17.503 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.501+0000 7f6c15305640 1 -- 172.21.15.64:0/3159961540 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c08001b70 con 0x7f6c10149ae0 2024-03-25T18:20:17.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.501+0000 7f6c15305640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:17.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.501+0000 7f6c15305640 1 -- 172.21.15.64:0/3159961540 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f6c08001f80 con 0x7f6c10149ae0 2024-03-25T18:20:17.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.501+0000 7f6c15305640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.501+0000 7f6c15305640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:17.504 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.501+0000 7f6c15305640 10 monclient: dump: 2024-03-25T18:20:17.504 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:17.504 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:17.504 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:17.504 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:17.504 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.505 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:17.505 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:17.505 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:17.505 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:17.505 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:17.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.501+0000 7f6c1856b640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:17.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.501+0000 7f6c1856b640 4 set_mon_vals no callback set 2024-03-25T18:20:17.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.502+0000 7f6c1856b640 10 monclient: shutdown 2024-03-25T18:20:17.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.502+0000 7f6c1856b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:17.507 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.502+0000 7f6c1856b640 1 -- 172.21.15.64:0/3159961540 >> v1:172.21.15.86:6789/0 conn(0x7f6c10149ae0 legacy=0x7f6c1014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.503+0000 7f6c1856b640 1 -- 172.21.15.64:0/3159961540 shutdown_connections 2024-03-25T18:20:17.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.503+0000 7f6c1856b640 1 -- 172.21.15.64:0/3159961540 >> 172.21.15.64:0/3159961540 conn(0x7f6c100b5d10 msgr2=0x7f6c100b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:17.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.503+0000 7f6c1856b640 1 -- 172.21.15.64:0/3159961540 shutdown_connections 2024-03-25T18:20:17.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.503+0000 7f6c1856b640 1 -- 172.21.15.64:0/3159961540 wait complete. 2024-03-25T18:20:17.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.504+0000 7f6c1856b640 10 monclient: build_initial_monmap 2024-03-25T18:20:17.508 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.504+0000 7f6c1856b640 10 monclient: monmap: 2024-03-25T18:20:17.508 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:20:17.508 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:17.508 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:20:17.508 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:20:17.508 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:17.508 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:20:17.509 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:20:17.509 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:20:17.509 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:20:17.509 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:17.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.504+0000 7f6c1856b640 1 librados: starting msgr at 2024-03-25T18:20:17.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.504+0000 7f6c1856b640 1 librados: starting objecter 2024-03-25T18:20:17.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.504+0000 7f6c1856b640 1 Processor -- start 2024-03-25T18:20:17.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.505+0000 7f6c1856b640 1 -- start start 2024-03-25T18:20:17.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.505+0000 7f6c1856b640 1 librados: setting wanted keys 2024-03-25T18:20:17.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.505+0000 7f6c1856b640 1 librados: calling monclient init 2024-03-25T18:20:17.509 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.505+0000 7f6c1856b640 10 monclient: init 2024-03-25T18:20:17.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.506+0000 7f6c1856b640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:17.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.506+0000 7f6c1856b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:17.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.506+0000 7f6c1856b640 10 monclient(hunting): picked mon.noname-a con 0x7f6c100705b0 addr v1:172.21.15.64:6789/0 2024-03-25T18:20:17.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.506+0000 7f6c1856b640 10 monclient(hunting): picked mon.noname-b con 0x7f6c101402d0 addr v1:172.21.15.86:6789/0 2024-03-25T18:20:17.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.506+0000 7f6c1856b640 10 monclient(hunting): picked mon.noname-c con 0x7f6c10149ae0 addr v1:172.21.15.64:6790/0 2024-03-25T18:20:17.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.506+0000 7f6c1856b640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c100aa3e0 con 0x7f6c100705b0 2024-03-25T18:20:17.511 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.506+0000 7f6c1856b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c101d9ad0 con 0x7f6c101402d0 2024-03-25T18:20:17.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.506+0000 7f6c1856b640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c100af5f0 con 0x7f6c10149ae0 2024-03-25T18:20:17.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.506+0000 7f6c1856b640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:17.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.506+0000 7f6c1856b640 10 monclient(hunting): authenticate will time out at 7603.745830s 2024-03-25T18:20:17.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.509+0000 7f6c16b08640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f6c10149ae0 0x7f6c10148410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:48894/0 (socket says 172.21.15.64:48894) 2024-03-25T18:20:17.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.509+0000 7f6c16b08640 1 -- 172.21.15.64:0/2671372671 learned_addr learned my addr 172.21.15.64:0/2671372671 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:20:17.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.510+0000 7f6c06ffd640 10 client.?.objecter ms_handle_connect 0x7f6c10149ae0 2024-03-25T18:20:17.512 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.510+0000 7f6c06ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:17.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.514+0000 7f6c06ffd640 10 client.?.objecter ms_handle_connect 0x7f6c101402d0 2024-03-25T18:20:17.516 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.514+0000 7f6c06ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:17.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.517+0000 7f6c06ffd640 10 client.?.objecter ms_handle_connect 0x7f6c100705b0 2024-03-25T18:20:17.519 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.517+0000 7f6c06ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:17.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.517+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953970174 0 0) 0x7f6c100af5f0 con 0x7f6c10149ae0 2024-03-25T18:20:17.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.517+0000 7f6c06ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.517+0000 7f6c06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.517+0000 7f6c06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.520 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.517+0000 7f6c06ffd640 10 monclient(hunting): my global_id is 4706 2024-03-25T18:20:17.521 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.517+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bf4004eb0 con 0x7f6c10149ae0 2024-03-25T18:20:17.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.524+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59340631 0 0) 0x7f6c101d9ad0 con 0x7f6c101402d0 2024-03-25T18:20:17.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.524+0000 7f6c06ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.524+0000 7f6c06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.524+0000 7f6c06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.524+0000 7f6c06ffd640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:20:17.527 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.524+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c100af5f0 con 0x7f6c101402d0 2024-03-25T18:20:17.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.529+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314278905 0 0) 0x7f6c100aa3e0 con 0x7f6c100705b0 2024-03-25T18:20:17.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.529+0000 7f6c06ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.529+0000 7f6c06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.531 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.529+0000 7f6c06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.529+0000 7f6c06ffd640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:20:17.532 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.529+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c101d9ad0 con 0x7f6c100705b0 2024-03-25T18:20:17.538 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.535+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1606871583 0 0) 0x7f6bf4004eb0 con 0x7f6c10149ae0 2024-03-25T18:20:17.538 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.535+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c100aa3e0 con 0x7f6c10149ae0 2024-03-25T18:20:17.538 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.535+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f6c08001e80 con 0x7f6c10149ae0 2024-03-25T18:20:17.538 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.535+0000 7f6c06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.538 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.535+0000 7f6c06ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:17.538 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.535+0000 7f6c06ffd640 10 monclient(hunting): dump: 2024-03-25T18:20:17.539 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:17.539 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:17.539 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:17.539 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:17.539 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.539 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:17.539 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:17.539 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:17.539 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:17.539 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:17.539 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.535+0000 7f6c06ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:17.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.541+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3960682574 0 0) 0x7f6c100af5f0 con 0x7f6c101402d0 2024-03-25T18:20:17.543 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.541+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bf4004eb0 con 0x7f6c101402d0 2024-03-25T18:20:17.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.541+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f6c000018b0 con 0x7f6c101402d0 2024-03-25T18:20:17.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.541+0000 7f6c06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.541+0000 7f6c06ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:17.544 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.541+0000 7f6c06ffd640 10 monclient(hunting): dump: 2024-03-25T18:20:17.544 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:17.544 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:17.544 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:17.544 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:17.544 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.544 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:17.544 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:17.544 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:17.545 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:17.545 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:17.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1805273870 0 0) 0x7f6bf4004eb0 con 0x7f6c101402d0 2024-03-25T18:20:17.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 >> v1:172.21.15.64:6790/0 conn(0x7f6c10149ae0 legacy=0x7f6c10148410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 >> v1:172.21.15.64:6789/0 conn(0x7f6c100705b0 legacy=0x7f6c101d91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c06ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:17.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c06ffd640 1 monclient: found mon.b 2024-03-25T18:20:17.554 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:17.555 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c06ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:17.555 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c100af780 con 0x7f6c101402d0 2024-03-25T18:20:17.555 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c06ffd640 10 monclient: _finish_auth 0 2024-03-25T18:20:17.555 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c06ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:17.556 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:17.556 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c1856b640 5 monclient: authenticate success, global_id 4711 2024-03-25T18:20:17.556 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c1856b640 10 monclient: _renew_subs 2024-03-25T18:20:17.556 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c1856b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:17.556 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c1856b640 1 -- 172.21.15.64:0/2671372671 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c1013a190 con 0x7f6c101402d0 2024-03-25T18:20:17.556 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c1856b640 10 client.4711.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:17.556 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c1856b640 10 monclient: _renew_subs 2024-03-25T18:20:17.556 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c1856b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:17.556 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c1856b640 1 -- 172.21.15.64:0/2671372671 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c10148b20 con 0x7f6c101402d0 2024-03-25T18:20:17.556 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.552+0000 7f6c1856b640 1 librados: init done 2024-03-25T18:20:17.556 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.553+0000 7f6c1856b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:17.556 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.553+0000 7f6c1856b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:17.557 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.553+0000 7f6c1856b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:17.557 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.553+0000 7f6c1856b640 1 -- 172.21.15.64:0/2671372671 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c101d9c60 con 0x7f6c101402d0 2024-03-25T18:20:17.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.848+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c00003610 con 0x7f6c101402d0 2024-03-25T18:20:17.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.848+0000 7f6c06ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:17.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.848+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f6c00003920 con 0x7f6c101402d0 2024-03-25T18:20:17.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.848+0000 7f6c06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.848+0000 7f6c06ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:17.850 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.848+0000 7f6c06ffd640 10 monclient: dump: 2024-03-25T18:20:17.850 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:17.850 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:17.850 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:17.850 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:17.851 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.851 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:17.851 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:17.851 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:17.851 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:17.851 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:17.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.848+0000 7f6c14b04640 4 set_mon_vals no callback set 2024-03-25T18:20:17.851 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.849+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f6c000198f0 con 0x7f6c101402d0 2024-03-25T18:20:17.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.849+0000 7f6c16307640 10 monclient: get_auth_request con 0x7f6bf406df70 auth_method 0 2024-03-25T18:20:17.852 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.850+0000 7f6c06ffd640 10 client.4711.objecter ms_handle_connect 0x7f6bf406df70 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.853+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4227240091 0 0) 0x7f6c00053590 con 0x7f6c101402d0 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.853+0000 7f6c06ffd640 10 client.4711.objecter ms_dispatch 0x7f6c10000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.853+0000 7f6c06ffd640 3 client.4711.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.853+0000 7f6c06ffd640 3 client.4711.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.853+0000 7f6c06ffd640 20 client.4711.objecter dump_active .. 0 homeless 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.854+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6c0004f0a0 con 0x7f6c101402d0 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.854+0000 7f6c06ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:17.854+0000 7f6c06ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:18.091 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.090+0000 7f6c1856b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:18.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.090+0000 7f6c1856b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:18.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.090+0000 7f6c1856b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:18.092 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.090+0000 7f6c1856b640 1 -- 172.21.15.64:0/2671372671 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c101d98e0 con 0x7f6c101402d0 2024-03-25T18:20:18.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.125+0000 7f6c06ffd640 1 -- 172.21.15.64:0/2671372671 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 991200065) 0x7f6c00050fd0 con 0x7f6c101402d0 2024-03-25T18:20:18.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.125+0000 7f6c06ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:18.126 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.125+0000 7f6c06ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:18.127 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:20:18.127 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:19:38.421162+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:38.421162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:18.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.127+0000 7f6c1856b640 10 librados: watch_flush enter 2024-03-25T18:20:18.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.127+0000 7f6c1856b640 10 librados: watch_flush exit 2024-03-25T18:20:18.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.127+0000 7f6c1856b640 20 client.4711.objecter shutdown clearing up homeless session... 2024-03-25T18:20:18.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.127+0000 7f6c1856b640 10 client.4711.objecter successfully canceled tick 2024-03-25T18:20:18.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.127+0000 7f6c1856b640 1 -- 172.21.15.64:0/2671372671 >> v1:172.21.15.64:6800/2915543638 conn(0x7f6bf406df70 legacy=0x7f6bf4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:18.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.127+0000 7f6c1856b640 10 monclient: shutdown 2024-03-25T18:20:18.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.127+0000 7f6c1856b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:18.130 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.127+0000 7f6c1856b640 1 -- 172.21.15.64:0/2671372671 >> v1:172.21.15.86:6789/0 conn(0x7f6c101402d0 legacy=0x7f6c100aeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:18.130 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.127+0000 7f6c1856b640 1 -- 172.21.15.64:0/2671372671 shutdown_connections 2024-03-25T18:20:18.130 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.127+0000 7f6c1856b640 1 -- 172.21.15.64:0/2671372671 >> 172.21.15.64:0/2671372671 conn(0x7f6c100b5d10 msgr2=0x7f6c100b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:18.130 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.127+0000 7f6c1856b640 1 -- 172.21.15.64:0/2671372671 shutdown_connections 2024-03-25T18:20:18.130 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.127+0000 7f6c1856b640 1 -- 172.21.15.64:0/2671372671 wait complete. 2024-03-25T18:20:18.130 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:18.127+0000 7f6c1856b640 1 librados: shutdown 2024-03-25T18:20:18.141 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:23.674 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:06.786 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:23.680 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:06.792 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:23.710 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:06.823 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:23.726 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:11:06.805 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:23.737 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:11:06.816 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:23.752 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:06.864 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:23.756 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:06.869 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:23.770 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:11:06.849 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:23.784 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:06.897 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:24.948 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:25.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.059+0000 7f29ab84d640 10 monclient: get_monmap_and_config 2024-03-25T18:20:25.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.059+0000 7f29ab84d640 10 monclient: build_initial_monmap 2024-03-25T18:20:25.060 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.059+0000 7f29ab84d640 10 monclient: monmap: 2024-03-25T18:20:25.061 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:20:25.061 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:25.061 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:20:25.060630+0000 2024-03-25T18:20:25.061 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:20:25.060630+0000 2024-03-25T18:20:25.061 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:25.061 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:25.061 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:20:25.061 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:20:25.061 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:20:25.061 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:25.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.060+0000 7f29ab84d640 1 Processor -- start 2024-03-25T18:20:25.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.060+0000 7f29ab84d640 1 -- start start 2024-03-25T18:20:25.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.060+0000 7f29ab84d640 10 monclient: init 2024-03-25T18:20:25.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.061+0000 7f29ab84d640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:25.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.061+0000 7f29ab84d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:25.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.061+0000 7f29ab84d640 10 monclient(hunting): picked mon.noname-c con 0x7f29a40b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:20:25.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.061+0000 7f29ab84d640 10 monclient(hunting): picked mon.noname-a con 0x7f29a40b1730 addr v1:172.21.15.64:6789/0 2024-03-25T18:20:25.062 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.061+0000 7f29ab84d640 10 monclient(hunting): picked mon.noname-b con 0x7f29a414be90 addr v1:172.21.15.64:6790/0 2024-03-25T18:20:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.061+0000 7f29ab84d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a40af940 con 0x7f29a40b1730 2024-03-25T18:20:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.061+0000 7f29ab84d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a40b0a70 con 0x7f29a40b4d60 2024-03-25T18:20:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.061+0000 7f29ab84d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a40b0d50 con 0x7f29a414be90 2024-03-25T18:20:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.061+0000 7f29ab84d640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:25.063 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.061+0000 7f29ab84d640 10 monclient(hunting): authenticate will time out at 7611.300046s 2024-03-25T18:20:25.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.064+0000 7f29a95e9640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f29a40b4d60 0x7f29a413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:39852/0 (socket says 172.21.15.64:39852) 2024-03-25T18:20:25.065 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.064+0000 7f29a95e9640 1 -- 172.21.15.64:0/1200667546 learned_addr learned my addr 172.21.15.64:0/1200667546 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:20:25.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.084+0000 7f299bfff640 1 -- 172.21.15.64:0/1200667546 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411739510 0 0) 0x7f29a40b0a70 con 0x7f29a40b4d60 2024-03-25T18:20:25.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.084+0000 7f299bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:25.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.084+0000 7f299bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:25.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.084+0000 7f299bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:25.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.084+0000 7f299bfff640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:20:25.085 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.084+0000 7f299bfff640 1 -- 172.21.15.64:0/1200667546 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2990003520 con 0x7f29a40b4d60 2024-03-25T18:20:25.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.098+0000 7f299bfff640 1 -- 172.21.15.64:0/1200667546 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726722006 0 0) 0x7f29a40b0d50 con 0x7f29a414be90 2024-03-25T18:20:25.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.098+0000 7f299bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:25.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.098+0000 7f299bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:25.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.098+0000 7f299bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:25.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.098+0000 7f299bfff640 10 monclient(hunting): my global_id is 4709 2024-03-25T18:20:25.099 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.098+0000 7f299bfff640 1 -- 172.21.15.64:0/1200667546 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29a40b0a70 con 0x7f29a414be90 2024-03-25T18:20:25.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.099+0000 7f299bfff640 1 -- 172.21.15.64:0/1200667546 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2234340225 0 0) 0x7f29a40af940 con 0x7f29a40b1730 2024-03-25T18:20:25.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.099+0000 7f299bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:25.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.099+0000 7f299bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:25.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.099+0000 7f299bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:25.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.099+0000 7f299bfff640 10 monclient(hunting): my global_id is 4704 2024-03-25T18:20:25.100 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.099+0000 7f299bfff640 1 -- 172.21.15.64:0/1200667546 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29a40b0d50 con 0x7f29a40b1730 2024-03-25T18:20:25.104 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.103+0000 7f299bfff640 1 -- 172.21.15.64:0/1200667546 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3795737608 0 0) 0x7f2990003520 con 0x7f29a40b4d60 2024-03-25T18:20:25.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.103+0000 7f299bfff640 1 -- 172.21.15.64:0/1200667546 >> v1:172.21.15.64:6790/0 conn(0x7f29a414be90 legacy=0x7f29a414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.103+0000 7f299bfff640 1 -- 172.21.15.64:0/1200667546 >> v1:172.21.15.64:6789/0 conn(0x7f29a40b1730 legacy=0x7f29a40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.103+0000 7f299bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:25.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.103+0000 7f299bfff640 1 monclient: found mon.noname-c 2024-03-25T18:20:25.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.103+0000 7f299bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:25.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.103+0000 7f299bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:20:25.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.103+0000 7f299bfff640 1 -- 172.21.15.64:0/1200667546 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a40b1030 con 0x7f29a40b4d60 2024-03-25T18:20:25.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.103+0000 7f299bfff640 10 monclient: _finish_auth 0 2024-03-25T18:20:25.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.103+0000 7f299bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:25.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.103+0000 7f299bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:25.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.103+0000 7f299bfff640 1 -- 172.21.15.64:0/1200667546 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f29a0001690 con 0x7f29a40b4d60 2024-03-25T18:20:25.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.103+0000 7f29ab84d640 5 monclient: authenticate success, global_id 4714 2024-03-25T18:20:25.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.103+0000 7f29ab84d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:25.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.103+0000 7f299bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:25.105 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.103+0000 7f299bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:25.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.103+0000 7f299bfff640 10 monclient: dump: 2024-03-25T18:20:25.106 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:25.106 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:25.106 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:25.106 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:25.106 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:25.106 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:25.106 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:25.106 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:25.106 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:25.106 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:25.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.103+0000 7f29ab84d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:25.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.118+0000 7f299bfff640 1 -- 172.21.15.64:0/1200667546 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29a00017f0 con 0x7f29a40b4d60 2024-03-25T18:20:25.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.118+0000 7f299bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:25.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.118+0000 7f299bfff640 1 -- 172.21.15.64:0/1200667546 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f29a0001c00 con 0x7f29a40b4d60 2024-03-25T18:20:25.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.118+0000 7f299bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:25.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.118+0000 7f299bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:25.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.118+0000 7f299bfff640 10 monclient: dump: 2024-03-25T18:20:25.119 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:25.119 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:25.119 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:25.119 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:25.120 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:25.120 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:25.120 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:25.120 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:25.120 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:25.120 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:25.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.118+0000 7f29ab84d640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:25.120 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.118+0000 7f29ab84d640 4 set_mon_vals no callback set 2024-03-25T18:20:25.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.118+0000 7f29ab84d640 10 monclient: shutdown 2024-03-25T18:20:25.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.118+0000 7f29ab84d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:25.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.118+0000 7f29ab84d640 1 -- 172.21.15.64:0/1200667546 >> v1:172.21.15.86:6789/0 conn(0x7f29a40b4d60 legacy=0x7f29a413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.118+0000 7f29ab84d640 1 -- 172.21.15.64:0/1200667546 shutdown_connections 2024-03-25T18:20:25.121 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.118+0000 7f29ab84d640 1 -- 172.21.15.64:0/1200667546 >> 172.21.15.64:0/1200667546 conn(0x7f29a40617a0 msgr2=0x7f29a4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:25.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.119+0000 7f29ab84d640 1 -- 172.21.15.64:0/1200667546 shutdown_connections 2024-03-25T18:20:25.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.119+0000 7f29ab84d640 1 -- 172.21.15.64:0/1200667546 wait complete. 2024-03-25T18:20:25.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.119+0000 7f29ab84d640 10 monclient: build_initial_monmap 2024-03-25T18:20:25.122 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.119+0000 7f29ab84d640 10 monclient: monmap: 2024-03-25T18:20:25.122 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:20:25.122 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:25.122 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:20:25.122 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:20:25.122 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:25.122 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:20:25.122 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:20:25.122 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:20:25.122 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:20:25.122 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:25.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.120+0000 7f29ab84d640 1 librados: starting msgr at 2024-03-25T18:20:25.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.120+0000 7f29ab84d640 1 librados: starting objecter 2024-03-25T18:20:25.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.120+0000 7f29ab84d640 1 Processor -- start 2024-03-25T18:20:25.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.120+0000 7f29ab84d640 1 -- start start 2024-03-25T18:20:25.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.120+0000 7f29ab84d640 1 librados: setting wanted keys 2024-03-25T18:20:25.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.120+0000 7f29ab84d640 1 librados: calling monclient init 2024-03-25T18:20:25.123 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.120+0000 7f29ab84d640 10 monclient: init 2024-03-25T18:20:25.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.121+0000 7f29ab84d640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:25.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.121+0000 7f29ab84d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:25.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.121+0000 7f29ab84d640 10 monclient(hunting): picked mon.noname-c con 0x7f29a40b1730 addr v1:172.21.15.64:6790/0 2024-03-25T18:20:25.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.121+0000 7f29ab84d640 10 monclient(hunting): picked mon.noname-b con 0x7f29a40b4d60 addr v1:172.21.15.86:6789/0 2024-03-25T18:20:25.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.121+0000 7f29ab84d640 10 monclient(hunting): picked mon.noname-a con 0x7f29a414be90 addr v1:172.21.15.64:6789/0 2024-03-25T18:20:25.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.121+0000 7f29ab84d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a4141f10 con 0x7f29a414be90 2024-03-25T18:20:25.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.121+0000 7f29ab84d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a41d7ab0 con 0x7f29a40b4d60 2024-03-25T18:20:25.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.121+0000 7f29ab84d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a41d86e0 con 0x7f29a40b1730 2024-03-25T18:20:25.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.121+0000 7f29ab84d640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:25.124 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.121+0000 7f29ab84d640 10 monclient(hunting): authenticate will time out at 7611.360057s 2024-03-25T18:20:25.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.128+0000 7f29a8de8640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f29a40b4d60 0x7f29a4142d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:39888/0 (socket says 172.21.15.64:39888) 2024-03-25T18:20:25.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.128+0000 7f29a8de8640 1 -- 172.21.15.64:0/440655561 learned_addr learned my addr 172.21.15.64:0/440655561 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:20:25.130 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.129+0000 7f2999ffb640 10 client.?.objecter ms_handle_connect 0x7f29a40b4d60 2024-03-25T18:20:25.130 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.129+0000 7f2999ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:25.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.136+0000 7f2999ffb640 10 client.?.objecter ms_handle_connect 0x7f29a414be90 2024-03-25T18:20:25.137 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.136+0000 7f2999ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:25.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.138+0000 7f2999ffb640 1 -- 172.21.15.64:0/440655561 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854894574 0 0) 0x7f29a41d7ab0 con 0x7f29a40b4d60 2024-03-25T18:20:25.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.138+0000 7f2999ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:25.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.138+0000 7f2999ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:25.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.138+0000 7f2999ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:25.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.138+0000 7f2999ffb640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:20:25.139 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.138+0000 7f2999ffb640 1 -- 172.21.15.64:0/440655561 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2988004eb0 con 0x7f29a40b4d60 2024-03-25T18:20:25.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.148+0000 7f2999ffb640 1 -- 172.21.15.64:0/440655561 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4036635853 0 0) 0x7f2988004eb0 con 0x7f29a40b4d60 2024-03-25T18:20:25.149 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.148+0000 7f2999ffb640 1 -- 172.21.15.64:0/440655561 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29a41d7ab0 con 0x7f29a40b4d60 2024-03-25T18:20:25.152 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.151+0000 7f2999ffb640 10 client.?.objecter ms_handle_connect 0x7f29a40b1730 2024-03-25T18:20:25.152 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.151+0000 7f2999ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:25.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.157+0000 7f2999ffb640 1 -- 172.21.15.64:0/440655561 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465062618 0 0) 0x7f29a4141f10 con 0x7f29a414be90 2024-03-25T18:20:25.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.157+0000 7f2999ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:25.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.157+0000 7f2999ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:25.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.157+0000 7f2999ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:25.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.157+0000 7f2999ffb640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:20:25.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.157+0000 7f2999ffb640 1 -- 172.21.15.64:0/440655561 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2988004eb0 con 0x7f29a414be90 2024-03-25T18:20:25.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.157+0000 7f2999ffb640 1 -- 172.21.15.64:0/440655561 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2994001b50 con 0x7f29a40b4d60 2024-03-25T18:20:25.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.157+0000 7f2999ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:25.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.157+0000 7f2999ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:25.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.157+0000 7f2999ffb640 10 monclient(hunting): dump: 2024-03-25T18:20:25.158 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:25.159 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:25.159 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:25.159 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:25.159 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:25.159 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:25.159 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:25.159 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:25.159 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:25.159 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:25.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.157+0000 7f2999ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:25.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f2999ffb640 1 -- 172.21.15.64:0/440655561 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2690209995 0 0) 0x7f29a41d7ab0 con 0x7f29a40b4d60 2024-03-25T18:20:25.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f2999ffb640 1 -- 172.21.15.64:0/440655561 >> v1:172.21.15.64:6790/0 conn(0x7f29a40b1730 legacy=0x7f29a41d7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.165 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f2999ffb640 1 -- 172.21.15.64:0/440655561 >> v1:172.21.15.64:6789/0 conn(0x7f29a414be90 legacy=0x7f29a41e31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f2999ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:20:25.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f2999ffb640 1 monclient: found mon.b 2024-03-25T18:20:25.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f2999ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:25.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f2999ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:25.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f2999ffb640 1 -- 172.21.15.64:0/440655561 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a41d8870 con 0x7f29a40b4d60 2024-03-25T18:20:25.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f2999ffb640 10 monclient: _finish_auth 0 2024-03-25T18:20:25.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f2999ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:20:25.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f2999ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:25.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f29ab84d640 5 monclient: authenticate success, global_id 4717 2024-03-25T18:20:25.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f29ab84d640 10 monclient: _renew_subs 2024-03-25T18:20:25.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f29ab84d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:25.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f29ab84d640 1 -- 172.21.15.64:0/440655561 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a4140db0 con 0x7f29a40b4d60 2024-03-25T18:20:25.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f29ab84d640 10 client.4717.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:25.166 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f29ab84d640 10 monclient: _renew_subs 2024-03-25T18:20:25.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f29ab84d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:25.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f29ab84d640 1 -- 172.21.15.64:0/440655561 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a4143640 con 0x7f29a40b4d60 2024-03-25T18:20:25.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.164+0000 7f29ab84d640 1 librados: init done 2024-03-25T18:20:25.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.165+0000 7f29ab84d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:25.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.165+0000 7f29ab84d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:25.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.165+0000 7f29ab84d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:25.167 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.165+0000 7f29ab84d640 1 -- 172.21.15.64:0/440655561 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a40627b0 con 0x7f29a40b4d60 2024-03-25T18:20:25.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.437+0000 7f2999ffb640 1 -- 172.21.15.64:0/440655561 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2994001cb0 con 0x7f29a40b4d60 2024-03-25T18:20:25.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.437+0000 7f2999ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:25.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.437+0000 7f2999ffb640 1 -- 172.21.15.64:0/440655561 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f2994003ae0 con 0x7f29a40b4d60 2024-03-25T18:20:25.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.437+0000 7f2999ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:25.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.437+0000 7f2999ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:25.438 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.437+0000 7f2999ffb640 10 monclient: dump: 2024-03-25T18:20:25.438 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:25.438 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:25.438 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:25.439 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:25.439 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:25.439 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:25.439 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:25.439 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:25.439 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:25.439 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:25.439 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.437+0000 7f299b7fe640 4 set_mon_vals no callback set 2024-03-25T18:20:25.439 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.437+0000 7f2999ffb640 1 -- 172.21.15.64:0/440655561 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f2994003d00 con 0x7f29a40b4d60 2024-03-25T18:20:25.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.438+0000 7f29a95e9640 10 monclient: get_auth_request con 0x7f298806dce0 auth_method 0 2024-03-25T18:20:25.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.438+0000 7f2999ffb640 10 client.4717.objecter ms_handle_connect 0x7f298806dce0 2024-03-25T18:20:25.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.441+0000 7f2999ffb640 1 -- 172.21.15.64:0/440655561 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4227240091 0 0) 0x7f29940549d0 con 0x7f29a40b4d60 2024-03-25T18:20:25.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.441+0000 7f2999ffb640 10 client.4717.objecter ms_dispatch 0x7f29a4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:20:25.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.441+0000 7f2999ffb640 3 client.4717.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:20:25.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.441+0000 7f2999ffb640 3 client.4717.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:20:25.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.442+0000 7f2999ffb640 20 client.4717.objecter dump_active .. 0 homeless 2024-03-25T18:20:25.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.442+0000 7f2999ffb640 1 -- 172.21.15.64:0/440655561 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f29940504e0 con 0x7f29a40b4d60 2024-03-25T18:20:25.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.442+0000 7f2999ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:25.443 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.442+0000 7f2999ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:25.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.697+0000 7f29ab84d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:25.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.697+0000 7f29ab84d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:25.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.697+0000 7f29ab84d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:25.698 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.697+0000 7f29ab84d640 1 -- 172.21.15.64:0/440655561 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29a4137ba0 con 0x7f29a40b4d60 2024-03-25T18:20:25.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.724+0000 7f2999ffb640 1 -- 172.21.15.64:0/440655561 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 991200065) 0x7f2994052410 con 0x7f29a40b4d60 2024-03-25T18:20:25.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.724+0000 7f2999ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:25.725 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.724+0000 7f2999ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:25.726 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:20:25.726 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:19:38.421162+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:38.421162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:25.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.727+0000 7f29ab84d640 10 librados: watch_flush enter 2024-03-25T18:20:25.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.727+0000 7f29ab84d640 10 librados: watch_flush exit 2024-03-25T18:20:25.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.727+0000 7f29ab84d640 20 client.4717.objecter shutdown clearing up homeless session... 2024-03-25T18:20:25.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.727+0000 7f29ab84d640 10 client.4717.objecter successfully canceled tick 2024-03-25T18:20:25.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.727+0000 7f29ab84d640 1 -- 172.21.15.64:0/440655561 >> v1:172.21.15.64:6800/2915543638 conn(0x7f298806dce0 legacy=0x7f29880701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.727+0000 7f29ab84d640 10 monclient: shutdown 2024-03-25T18:20:25.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.727+0000 7f29ab84d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:25.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.727+0000 7f29ab84d640 1 -- 172.21.15.64:0/440655561 >> v1:172.21.15.86:6789/0 conn(0x7f29a40b4d60 legacy=0x7f29a4142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.727+0000 7f29ab84d640 1 -- 172.21.15.64:0/440655561 shutdown_connections 2024-03-25T18:20:25.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.727+0000 7f29ab84d640 1 -- 172.21.15.64:0/440655561 >> 172.21.15.64:0/440655561 conn(0x7f29a40617a0 msgr2=0x7f29a40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:25.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.727+0000 7f29ab84d640 1 -- 172.21.15.64:0/440655561 shutdown_connections 2024-03-25T18:20:25.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.727+0000 7f29ab84d640 1 -- 172.21.15.64:0/440655561 wait complete. 2024-03-25T18:20:25.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:25.727+0000 7f29ab84d640 1 librados: shutdown 2024-03-25T18:20:25.742 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:32.550 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:32.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.663+0000 7fb9b30a8640 10 monclient: get_monmap_and_config 2024-03-25T18:20:32.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.663+0000 7fb9b30a8640 10 monclient: build_initial_monmap 2024-03-25T18:20:32.664 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.663+0000 7fb9b30a8640 10 monclient: monmap: 2024-03-25T18:20:32.664 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:20:32.664 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:32.664 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:20:32.664217+0000 2024-03-25T18:20:32.664 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:20:32.664217+0000 2024-03-25T18:20:32.665 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:32.665 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:32.665 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:20:32.665 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:20:32.665 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:20:32.665 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:32.666 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.664+0000 7fb9b30a8640 1 Processor -- start 2024-03-25T18:20:32.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.664+0000 7fb9b30a8640 1 -- start start 2024-03-25T18:20:32.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.664+0000 7fb9b30a8640 10 monclient: init 2024-03-25T18:20:32.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.664+0000 7fb9b30a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:32.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.664+0000 7fb9b30a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:32.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.664+0000 7fb9b30a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb9ac0a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:20:32.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.664+0000 7fb9b30a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb9ac13f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:20:32.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.664+0000 7fb9b30a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb9ac1430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:20:32.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.664+0000 7fb9b30a8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac142520 con 0x7fb9ac1430c0 2024-03-25T18:20:32.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.664+0000 7fb9b30a8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac0b28a0 con 0x7fb9ac13f710 2024-03-25T18:20:32.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.664+0000 7fb9b30a8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac0b2a30 con 0x7fb9ac0a98f0 2024-03-25T18:20:32.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.664+0000 7fb9b30a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:32.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.664+0000 7fb9b30a8640 10 monclient(hunting): authenticate will time out at 7618.903361s 2024-03-25T18:20:32.676 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.676+0000 7fb9b0e44640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7fb9ac0a98f0 0x7fb9ac0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:59340/0 (socket says 172.21.15.64:59340) 2024-03-25T18:20:32.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.676+0000 7fb9b0e44640 1 -- 172.21.15.64:0/3798315358 learned_addr learned my addr 172.21.15.64:0/3798315358 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:20:32.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.687+0000 7fb9ab7fe640 1 -- 172.21.15.64:0/3798315358 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986286281 0 0) 0x7fb9ac0b2a30 con 0x7fb9ac0a98f0 2024-03-25T18:20:32.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.687+0000 7fb9ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:32.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.687+0000 7fb9ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:32.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.687+0000 7fb9ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:32.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.687+0000 7fb9ab7fe640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:20:32.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.687+0000 7fb9ab7fe640 1 -- 172.21.15.64:0/3798315358 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb99c003520 con 0x7fb9ac0a98f0 2024-03-25T18:20:32.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.699+0000 7fb9ab7fe640 1 -- 172.21.15.64:0/3798315358 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178519852 0 0) 0x7fb9ac142520 con 0x7fb9ac1430c0 2024-03-25T18:20:32.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.699+0000 7fb9ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:32.700 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.699+0000 7fb9ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:32.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.699+0000 7fb9ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:32.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.699+0000 7fb9ab7fe640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:20:32.701 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.699+0000 7fb9ab7fe640 1 -- 172.21.15.64:0/3798315358 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9ac0b2a30 con 0x7fb9ac1430c0 2024-03-25T18:20:32.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.708+0000 7fb9ab7fe640 1 -- 172.21.15.64:0/3798315358 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3286963212 0 0) 0x7fb99c003520 con 0x7fb9ac0a98f0 2024-03-25T18:20:32.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.708+0000 7fb9ab7fe640 1 -- 172.21.15.64:0/3798315358 >> v1:172.21.15.86:6789/0 conn(0x7fb9ac13f710 legacy=0x7fb9ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.708+0000 7fb9ab7fe640 1 -- 172.21.15.64:0/3798315358 >> v1:172.21.15.64:6789/0 conn(0x7fb9ac1430c0 legacy=0x7fb9ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.708+0000 7fb9ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:32.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.708+0000 7fb9ab7fe640 1 monclient: found mon.noname-b 2024-03-25T18:20:32.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.708+0000 7fb9ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:32.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.708+0000 7fb9ab7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:20:32.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.708+0000 7fb9ab7fe640 1 -- 172.21.15.64:0/3798315358 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ac0b44c0 con 0x7fb9ac0a98f0 2024-03-25T18:20:32.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.708+0000 7fb9ab7fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:32.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.708+0000 7fb9ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:32.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.709+0000 7fb9ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:32.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.709+0000 7fb9ab7fe640 1 -- 172.21.15.64:0/3798315358 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb9a0001690 con 0x7fb9ac0a98f0 2024-03-25T18:20:32.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.709+0000 7fb9b30a8640 5 monclient: authenticate success, global_id 4715 2024-03-25T18:20:32.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.709+0000 7fb9b30a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:32.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.709+0000 7fb9ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:32.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.709+0000 7fb9ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:32.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.709+0000 7fb9ab7fe640 10 monclient: dump: 2024-03-25T18:20:32.711 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:32.711 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:32.711 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:32.711 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:32.712 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:32.712 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:32.712 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:32.712 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:32.712 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:32.712 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:32.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.709+0000 7fb9b30a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:32.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.740+0000 7fb9ab7fe640 1 -- 172.21.15.64:0/3798315358 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9a00017f0 con 0x7fb9ac0a98f0 2024-03-25T18:20:32.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.740+0000 7fb9ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:32.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.740+0000 7fb9ab7fe640 1 -- 172.21.15.64:0/3798315358 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb9a0001c00 con 0x7fb9ac0a98f0 2024-03-25T18:20:32.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.740+0000 7fb9ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:32.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.740+0000 7fb9ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:32.742 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.740+0000 7fb9ab7fe640 10 monclient: dump: 2024-03-25T18:20:32.742 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:32.742 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:32.743 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:32.743 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:32.743 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:32.743 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:32.743 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:32.743 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:32.743 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:32.743 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:32.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.740+0000 7fb9b30a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:32.743 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.741+0000 7fb9b30a8640 4 set_mon_vals no callback set 2024-03-25T18:20:32.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.741+0000 7fb9b30a8640 10 monclient: shutdown 2024-03-25T18:20:32.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.741+0000 7fb9b30a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:32.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.741+0000 7fb9b30a8640 1 -- 172.21.15.64:0/3798315358 >> v1:172.21.15.64:6790/0 conn(0x7fb9ac0a98f0 legacy=0x7fb9ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.741+0000 7fb9b30a8640 1 -- 172.21.15.64:0/3798315358 shutdown_connections 2024-03-25T18:20:32.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.741+0000 7fb9b30a8640 1 -- 172.21.15.64:0/3798315358 >> 172.21.15.64:0/3798315358 conn(0x7fb9ac0b7ea0 msgr2=0x7fb9ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:32.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.742+0000 7fb9b30a8640 1 -- 172.21.15.64:0/3798315358 shutdown_connections 2024-03-25T18:20:32.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.742+0000 7fb9b30a8640 1 -- 172.21.15.64:0/3798315358 wait complete. 2024-03-25T18:20:32.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.743+0000 7fb9b30a8640 10 monclient: build_initial_monmap 2024-03-25T18:20:32.746 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.743+0000 7fb9b30a8640 10 monclient: monmap: 2024-03-25T18:20:32.746 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:20:32.746 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:32.747 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:20:32.747 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:20:32.747 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:32.747 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:20:32.747 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:20:32.747 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:20:32.747 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:20:32.747 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:32.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.743+0000 7fb9b30a8640 1 librados: starting msgr at 2024-03-25T18:20:32.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.743+0000 7fb9b30a8640 1 librados: starting objecter 2024-03-25T18:20:32.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.743+0000 7fb9b30a8640 1 Processor -- start 2024-03-25T18:20:32.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.743+0000 7fb9b30a8640 1 -- start start 2024-03-25T18:20:32.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.743+0000 7fb9b30a8640 1 librados: setting wanted keys 2024-03-25T18:20:32.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.743+0000 7fb9b30a8640 1 librados: calling monclient init 2024-03-25T18:20:32.747 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.743+0000 7fb9b30a8640 10 monclient: init 2024-03-25T18:20:32.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.744+0000 7fb9b30a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:32.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.744+0000 7fb9b30a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:32.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.744+0000 7fb9b30a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb9ac0a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:20:32.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.744+0000 7fb9b30a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb9ac13f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:20:32.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.744+0000 7fb9b30a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb9ac1430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:20:32.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.744+0000 7fb9b30a8640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac1370a0 con 0x7fb9ac0a98f0 2024-03-25T18:20:32.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.745+0000 7fb9b30a8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac135090 con 0x7fb9ac13f710 2024-03-25T18:20:32.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.745+0000 7fb9b30a8640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ac135220 con 0x7fb9ac1430c0 2024-03-25T18:20:32.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.745+0000 7fb9b30a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:32.748 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.745+0000 7fb9b30a8640 10 monclient(hunting): authenticate will time out at 7618.983526s 2024-03-25T18:20:32.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.747+0000 7fb9abfff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb9ac13f710 0x7fb9ac1d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:49194/0 (socket says 172.21.15.64:49194) 2024-03-25T18:20:32.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.747+0000 7fb9abfff640 1 -- 172.21.15.64:0/3319694365 learned_addr learned my addr 172.21.15.64:0/3319694365 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:20:32.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.748+0000 7fb9a97fa640 10 client.?.objecter ms_handle_connect 0x7fb9ac13f710 2024-03-25T18:20:32.749 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.748+0000 7fb9a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:32.754 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.754+0000 7fb9a97fa640 10 client.?.objecter ms_handle_connect 0x7fb9ac0a98f0 2024-03-25T18:20:32.755 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.754+0000 7fb9a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:32.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.760+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369255087 0 0) 0x7fb9ac135090 con 0x7fb9ac13f710 2024-03-25T18:20:32.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.760+0000 7fb9a97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:32.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.760+0000 7fb9a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:32.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.760+0000 7fb9a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:32.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.760+0000 7fb9a97fa640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:20:32.761 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.760+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb990004eb0 con 0x7fb9ac13f710 2024-03-25T18:20:32.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.761+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745451343 0 0) 0x7fb9ac1370a0 con 0x7fb9ac0a98f0 2024-03-25T18:20:32.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.761+0000 7fb9a97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:32.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.761+0000 7fb9a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:32.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.761+0000 7fb9a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:32.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.761+0000 7fb9a97fa640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:20:32.762 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.761+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9ac135090 con 0x7fb9ac0a98f0 2024-03-25T18:20:32.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.763+0000 7fb9a97fa640 10 client.?.objecter ms_handle_connect 0x7fb9ac1430c0 2024-03-25T18:20:32.764 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.763+0000 7fb9a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:32.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.787+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 761256178 0 0) 0x7fb990004eb0 con 0x7fb9ac13f710 2024-03-25T18:20:32.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.787+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9ac1370a0 con 0x7fb9ac13f710 2024-03-25T18:20:32.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.787+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb9980019d0 con 0x7fb9ac13f710 2024-03-25T18:20:32.788 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.787+0000 7fb9a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:32.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.787+0000 7fb9a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:32.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.787+0000 7fb9a97fa640 10 monclient(hunting): dump: 2024-03-25T18:20:32.789 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:32.789 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:32.789 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:32.789 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:32.789 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:32.789 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:32.789 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:32.789 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:32.789 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:32.789 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:32.789 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.787+0000 7fb9a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:32.791 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.790+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585155415 0 0) 0x7fb9ac135220 con 0x7fb9ac1430c0 2024-03-25T18:20:32.791 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.790+0000 7fb9a97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:32.791 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.790+0000 7fb9a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:32.791 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.790+0000 7fb9a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:32.791 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.790+0000 7fb9a97fa640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:20:32.791 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.790+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb990004eb0 con 0x7fb9ac1430c0 2024-03-25T18:20:32.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.799+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3330071103 0 0) 0x7fb9ac135090 con 0x7fb9ac0a98f0 2024-03-25T18:20:32.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.799+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9ac135220 con 0x7fb9ac0a98f0 2024-03-25T18:20:32.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.799+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb9a0001b00 con 0x7fb9ac0a98f0 2024-03-25T18:20:32.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.799+0000 7fb9a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:32.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.799+0000 7fb9a97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:32.800 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.799+0000 7fb9a97fa640 10 monclient(hunting): dump: 2024-03-25T18:20:32.801 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:32.801 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:32.801 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:32.801 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:32.801 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:32.801 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:32.801 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:32.801 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:32.801 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:32.801 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:32.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.800+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1067713980 0 0) 0x7fb9ac1370a0 con 0x7fb9ac13f710 2024-03-25T18:20:32.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 >> v1:172.21.15.64:6790/0 conn(0x7fb9ac1430c0 legacy=0x7fb9ac1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 >> v1:172.21.15.64:6789/0 conn(0x7fb9ac0a98f0 legacy=0x7fb9ac1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9a97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:20:32.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9a97fa640 1 monclient: found mon.b 2024-03-25T18:20:32.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:32.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9a97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:32.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ac1d5000 con 0x7fb9ac13f710 2024-03-25T18:20:32.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9a97fa640 10 monclient: _finish_auth 0 2024-03-25T18:20:32.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9a97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:20:32.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:32.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9b30a8640 5 monclient: authenticate success, global_id 4723 2024-03-25T18:20:32.802 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9b30a8640 10 monclient: _renew_subs 2024-03-25T18:20:32.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9b30a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:32.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9b30a8640 1 -- 172.21.15.64:0/3319694365 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9ac1d52b0 con 0x7fb9ac13f710 2024-03-25T18:20:32.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9b30a8640 10 client.4723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:32.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9b30a8640 10 monclient: _renew_subs 2024-03-25T18:20:32.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9b30a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:32.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9b30a8640 1 -- 172.21.15.64:0/3319694365 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9ac135910 con 0x7fb9ac13f710 2024-03-25T18:20:32.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.801+0000 7fb9b30a8640 1 librados: init done 2024-03-25T18:20:32.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.802+0000 7fb9b30a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:32.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.802+0000 7fb9b30a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:32.803 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.802+0000 7fb9b30a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:32.804 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:32.802+0000 7fb9b30a8640 1 -- 172.21.15.64:0/3319694365 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9ac0627b0 con 0x7fb9ac13f710 2024-03-25T18:20:33.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.069+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb998003610 con 0x7fb9ac13f710 2024-03-25T18:20:33.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.069+0000 7fb9a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:33.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.069+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7fb998003a20 con 0x7fb9ac13f710 2024-03-25T18:20:33.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.069+0000 7fb9a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:33.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.069+0000 7fb9a97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:33.070 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.069+0000 7fb9a97fa640 10 monclient: dump: 2024-03-25T18:20:33.071 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:33.071 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:33.071 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:33.071 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:33.071 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:33.071 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:33.071 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:33.071 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:33.071 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:33.071 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:33.071 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.069+0000 7fb9aaffd640 4 set_mon_vals no callback set 2024-03-25T18:20:33.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.070+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7fb998003c40 con 0x7fb9ac13f710 2024-03-25T18:20:33.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.070+0000 7fb9b0e44640 10 monclient: get_auth_request con 0x7fb99006dfb0 auth_method 0 2024-03-25T18:20:33.072 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.070+0000 7fb9a97fa640 10 client.4723.objecter ms_handle_connect 0x7fb99006dfb0 2024-03-25T18:20:33.073 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.072+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4227240091 0 0) 0x7fb998054790 con 0x7fb9ac13f710 2024-03-25T18:20:33.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.072+0000 7fb9a97fa640 10 client.4723.objecter ms_dispatch 0x7fb9ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:20:33.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.073+0000 7fb9a97fa640 3 client.4723.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:20:33.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.073+0000 7fb9a97fa640 3 client.4723.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:20:33.074 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.073+0000 7fb9a97fa640 20 client.4723.objecter dump_active .. 0 homeless 2024-03-25T18:20:33.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.076+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb9980502a0 con 0x7fb9ac13f710 2024-03-25T18:20:33.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.076+0000 7fb9a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:33.077 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.076+0000 7fb9a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:33.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.333+0000 7fb9b30a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:33.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.333+0000 7fb9b30a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:33.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.333+0000 7fb9b30a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-25T18:20:33.334 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.333+0000 7fb9b30a8640 1 -- 172.21.15.64:0/3319694365 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9ac13bfe0 con 0x7fb9ac13f710 2024-03-25T18:20:33.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.361+0000 7fb9a97fa640 1 -- 172.21.15.64:0/3319694365 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 991200065) 0x7fb9980521d0 con 0x7fb9ac13f710 2024-03-25T18:20:33.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.361+0000 7fb9a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:33.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.361+0000 7fb9a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:33.363 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:20:33.363 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:19:38.421162+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:38.421162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:33.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.364+0000 7fb9b30a8640 10 librados: watch_flush enter 2024-03-25T18:20:33.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.364+0000 7fb9b30a8640 10 librados: watch_flush exit 2024-03-25T18:20:33.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.364+0000 7fb9b30a8640 20 client.4723.objecter shutdown clearing up homeless session... 2024-03-25T18:20:33.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.364+0000 7fb9b30a8640 10 client.4723.objecter successfully canceled tick 2024-03-25T18:20:33.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.364+0000 7fb9b30a8640 1 -- 172.21.15.64:0/3319694365 >> v1:172.21.15.64:6800/2915543638 conn(0x7fb99006dfb0 legacy=0x7fb990070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.364+0000 7fb9b30a8640 10 monclient: shutdown 2024-03-25T18:20:33.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.364+0000 7fb9b30a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:33.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.364+0000 7fb9b30a8640 1 -- 172.21.15.64:0/3319694365 >> v1:172.21.15.86:6789/0 conn(0x7fb9ac13f710 legacy=0x7fb9ac1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.364+0000 7fb9b30a8640 1 -- 172.21.15.64:0/3319694365 shutdown_connections 2024-03-25T18:20:33.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.364+0000 7fb9b30a8640 1 -- 172.21.15.64:0/3319694365 >> 172.21.15.64:0/3319694365 conn(0x7fb9ac0b7ea0 msgr2=0x7fb9ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:33.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.364+0000 7fb9b30a8640 1 -- 172.21.15.64:0/3319694365 shutdown_connections 2024-03-25T18:20:33.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.364+0000 7fb9b30a8640 1 -- 172.21.15.64:0/3319694365 wait complete. 2024-03-25T18:20:33.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:33.364+0000 7fb9b30a8640 1 librados: shutdown 2024-03-25T18:20:33.376 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:40.182 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:40.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.293+0000 7ffac28ba640 10 monclient: get_monmap_and_config 2024-03-25T18:20:40.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.293+0000 7ffac28ba640 10 monclient: build_initial_monmap 2024-03-25T18:20:40.294 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.293+0000 7ffac28ba640 10 monclient: monmap: 2024-03-25T18:20:40.294 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:20:40.294 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:40.295 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:20:40.294585+0000 2024-03-25T18:20:40.295 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:20:40.294585+0000 2024-03-25T18:20:40.295 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:40.295 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:40.295 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:20:40.295 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:20:40.295 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:20:40.295 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:40.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.294+0000 7ffac28ba640 1 Processor -- start 2024-03-25T18:20:40.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.294+0000 7ffac28ba640 1 -- start start 2024-03-25T18:20:40.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.294+0000 7ffac28ba640 10 monclient: init 2024-03-25T18:20:40.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.295+0000 7ffac28ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:40.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.295+0000 7ffac28ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:40.296 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.295+0000 7ffac28ba640 10 monclient(hunting): picked mon.noname-c con 0x7ffabc0a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:20:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.295+0000 7ffac28ba640 10 monclient(hunting): picked mon.noname-a con 0x7ffabc13f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:20:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.295+0000 7ffac28ba640 10 monclient(hunting): picked mon.noname-b con 0x7ffabc1430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:20:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.295+0000 7ffac28ba640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc142520 con 0x7ffabc13f710 2024-03-25T18:20:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.295+0000 7ffac28ba640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc0b28a0 con 0x7ffabc0a98f0 2024-03-25T18:20:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.295+0000 7ffac28ba640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc0b2a30 con 0x7ffabc1430c0 2024-03-25T18:20:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.295+0000 7ffac28ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:40.297 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.295+0000 7ffac28ba640 10 monclient(hunting): authenticate will time out at 7626.533850s 2024-03-25T18:20:40.315 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.315+0000 7ffabbfff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ffabc0a98f0 0x7ffabc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:34088/0 (socket says 172.21.15.64:34088) 2024-03-25T18:20:40.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.315+0000 7ffabbfff640 1 -- 172.21.15.64:0/88388524 learned_addr learned my addr 172.21.15.64:0/88388524 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:20:40.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.334+0000 7ffabaffd640 1 -- 172.21.15.64:0/88388524 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4096335958 0 0) 0x7ffabc142520 con 0x7ffabc13f710 2024-03-25T18:20:40.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.334+0000 7ffabaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:40.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.334+0000 7ffabaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:40.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.334+0000 7ffabaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:40.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.334+0000 7ffabaffd640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:20:40.335 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.334+0000 7ffabaffd640 1 -- 172.21.15.64:0/88388524 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa4003520 con 0x7ffabc13f710 2024-03-25T18:20:40.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.337+0000 7ffabaffd640 1 -- 172.21.15.64:0/88388524 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789646398 0 0) 0x7ffabc0b28a0 con 0x7ffabc0a98f0 2024-03-25T18:20:40.338 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.337+0000 7ffabaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:40.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.337+0000 7ffabaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:40.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.337+0000 7ffabaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:40.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.337+0000 7ffabaffd640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:20:40.339 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.337+0000 7ffabaffd640 1 -- 172.21.15.64:0/88388524 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffabc142520 con 0x7ffabc0a98f0 2024-03-25T18:20:40.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.341+0000 7ffabaffd640 1 -- 172.21.15.64:0/88388524 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212635890 0 0) 0x7ffabc0b2a30 con 0x7ffabc1430c0 2024-03-25T18:20:40.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.341+0000 7ffabaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:40.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.341+0000 7ffabaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:40.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.341+0000 7ffabaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:40.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.341+0000 7ffabaffd640 10 monclient(hunting): my global_id is 4721 2024-03-25T18:20:40.342 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.341+0000 7ffabaffd640 1 -- 172.21.15.64:0/88388524 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffabc0b28a0 con 0x7ffabc1430c0 2024-03-25T18:20:40.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.353+0000 7ffabaffd640 1 -- 172.21.15.64:0/88388524 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2711412534 0 0) 0x7ffabc142520 con 0x7ffabc0a98f0 2024-03-25T18:20:40.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.353+0000 7ffabaffd640 1 -- 172.21.15.64:0/88388524 >> v1:172.21.15.64:6790/0 conn(0x7ffabc1430c0 legacy=0x7ffabc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.353+0000 7ffabaffd640 1 -- 172.21.15.64:0/88388524 >> v1:172.21.15.64:6789/0 conn(0x7ffabc13f710 legacy=0x7ffabc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.353+0000 7ffabaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:40.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.353+0000 7ffabaffd640 1 monclient: found mon.noname-c 2024-03-25T18:20:40.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.353+0000 7ffabaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:40.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.353+0000 7ffabaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-25T18:20:40.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.353+0000 7ffabaffd640 1 -- 172.21.15.64:0/88388524 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffabc0b44c0 con 0x7ffabc0a98f0 2024-03-25T18:20:40.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.353+0000 7ffabaffd640 10 monclient: _finish_auth 0 2024-03-25T18:20:40.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.353+0000 7ffabaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:40.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.353+0000 7ffabaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:40.355 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.353+0000 7ffac28ba640 5 monclient: authenticate success, global_id 4726 2024-03-25T18:20:40.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.353+0000 7ffac28ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:40.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.353+0000 7ffabaffd640 1 -- 172.21.15.64:0/88388524 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ffaac001670 con 0x7ffabc0a98f0 2024-03-25T18:20:40.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.353+0000 7ffabaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.353+0000 7ffabaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:40.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.353+0000 7ffabaffd640 10 monclient: dump: 2024-03-25T18:20:40.356 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:40.356 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:40.356 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:40.356 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:40.356 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.356 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:40.356 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:40.356 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:40.356 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:40.356 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:40.356 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.353+0000 7ffac28ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:40.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.360+0000 7ffabaffd640 1 -- 172.21.15.64:0/88388524 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffaac0017d0 con 0x7ffabc0a98f0 2024-03-25T18:20:40.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.360+0000 7ffabaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:40.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.360+0000 7ffabaffd640 1 -- 172.21.15.64:0/88388524 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ffaac001be0 con 0x7ffabc0a98f0 2024-03-25T18:20:40.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.360+0000 7ffabaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.360+0000 7ffabaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:40.361 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.360+0000 7ffabaffd640 10 monclient: dump: 2024-03-25T18:20:40.361 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:40.361 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:40.361 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:40.361 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:40.361 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.361 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:40.362 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:40.362 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:40.362 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:40.362 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:40.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.360+0000 7ffac28ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:40.362 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.360+0000 7ffac28ba640 4 set_mon_vals no callback set 2024-03-25T18:20:40.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.361+0000 7ffac28ba640 10 monclient: shutdown 2024-03-25T18:20:40.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.361+0000 7ffac28ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:40.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.361+0000 7ffac28ba640 1 -- 172.21.15.64:0/88388524 >> v1:172.21.15.86:6789/0 conn(0x7ffabc0a98f0 legacy=0x7ffabc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.361+0000 7ffac28ba640 1 -- 172.21.15.64:0/88388524 shutdown_connections 2024-03-25T18:20:40.363 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.361+0000 7ffac28ba640 1 -- 172.21.15.64:0/88388524 >> 172.21.15.64:0/88388524 conn(0x7ffabc0b7ea0 msgr2=0x7ffabc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:40.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.361+0000 7ffac28ba640 1 -- 172.21.15.64:0/88388524 shutdown_connections 2024-03-25T18:20:40.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.361+0000 7ffac28ba640 1 -- 172.21.15.64:0/88388524 wait complete. 2024-03-25T18:20:40.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.362+0000 7ffac28ba640 10 monclient: build_initial_monmap 2024-03-25T18:20:40.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.362+0000 7ffac28ba640 10 monclient: monmap: 2024-03-25T18:20:40.364 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:20:40.364 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:40.364 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:20:40.364 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:20:40.364 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:40.364 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:20:40.364 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:20:40.364 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:20:40.364 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:20:40.364 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:40.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.362+0000 7ffac28ba640 1 librados: starting msgr at 2024-03-25T18:20:40.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.362+0000 7ffac28ba640 1 librados: starting objecter 2024-03-25T18:20:40.364 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.362+0000 7ffac28ba640 1 Processor -- start 2024-03-25T18:20:40.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.362+0000 7ffac28ba640 1 -- start start 2024-03-25T18:20:40.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.362+0000 7ffac28ba640 1 librados: setting wanted keys 2024-03-25T18:20:40.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.362+0000 7ffac28ba640 1 librados: calling monclient init 2024-03-25T18:20:40.365 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.362+0000 7ffac28ba640 10 monclient: init 2024-03-25T18:20:40.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.363+0000 7ffac28ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:40.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.363+0000 7ffac28ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:40.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.363+0000 7ffac28ba640 10 monclient(hunting): picked mon.noname-b con 0x7ffabc0a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:20:40.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.363+0000 7ffac28ba640 10 monclient(hunting): picked mon.noname-a con 0x7ffabc13f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:20:40.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.363+0000 7ffac28ba640 10 monclient(hunting): picked mon.noname-c con 0x7ffabc1430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:20:40.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.363+0000 7ffac28ba640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc137f50 con 0x7ffabc13f710 2024-03-25T18:20:40.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.363+0000 7ffac28ba640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc139830 con 0x7ffabc0a98f0 2024-03-25T18:20:40.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.363+0000 7ffac28ba640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc0aba50 con 0x7ffabc1430c0 2024-03-25T18:20:40.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.363+0000 7ffac28ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:40.366 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.363+0000 7ffac28ba640 10 monclient(hunting): authenticate will time out at 7626.602125s 2024-03-25T18:20:40.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.374+0000 7ffabbfff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ffabc0a98f0 0x7ffabc134080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:34152/0 (socket says 172.21.15.64:34152) 2024-03-25T18:20:40.375 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.374+0000 7ffabbfff640 1 -- 172.21.15.64:0/1189310368 learned_addr learned my addr 172.21.15.64:0/1189310368 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:20:40.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.375+0000 7ffab8ff9640 10 client.?.objecter ms_handle_connect 0x7ffabc0a98f0 2024-03-25T18:20:40.376 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.375+0000 7ffab8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:40.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.376+0000 7ffab8ff9640 10 client.?.objecter ms_handle_connect 0x7ffabc13f710 2024-03-25T18:20:40.377 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.376+0000 7ffab8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:40.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.386+0000 7ffab8ff9640 10 client.?.objecter ms_handle_connect 0x7ffabc1430c0 2024-03-25T18:20:40.387 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.386+0000 7ffab8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:40.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.389+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975852935 0 0) 0x7ffabc137f50 con 0x7ffabc13f710 2024-03-25T18:20:40.390 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.389+0000 7ffab8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:40.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.389+0000 7ffab8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:40.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.389+0000 7ffab8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:40.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.389+0000 7ffab8ff9640 10 monclient(hunting): my global_id is 4719 2024-03-25T18:20:40.391 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.389+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa8004eb0 con 0x7ffabc13f710 2024-03-25T18:20:40.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.391+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179173692 0 0) 0x7ffabc139830 con 0x7ffabc0a98f0 2024-03-25T18:20:40.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.391+0000 7ffab8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:40.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.391+0000 7ffab8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:40.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.391+0000 7ffab8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:40.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.391+0000 7ffab8ff9640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:20:40.392 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.391+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffabc137f50 con 0x7ffabc0a98f0 2024-03-25T18:20:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.398+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3410980557 0 0) 0x7ffaa8004eb0 con 0x7ffabc13f710 2024-03-25T18:20:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.398+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffabc139830 con 0x7ffabc13f710 2024-03-25T18:20:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.398+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ffab0001950 con 0x7ffabc13f710 2024-03-25T18:20:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.398+0000 7ffab8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.398+0000 7ffab8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:40.399 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.398+0000 7ffab8ff9640 10 monclient(hunting): dump: 2024-03-25T18:20:40.399 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:40.399 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:40.399 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:40.399 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:40.399 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.399 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:40.399 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:40.399 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:40.399 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:40.400 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:40.400 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.398+0000 7ffab8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:40.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.402+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 35306743 0 0) 0x7ffabc137f50 con 0x7ffabc0a98f0 2024-03-25T18:20:40.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.402+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffaa8004eb0 con 0x7ffabc0a98f0 2024-03-25T18:20:40.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.402+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ffaac001ba0 con 0x7ffabc0a98f0 2024-03-25T18:20:40.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.402+0000 7ffab8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.402+0000 7ffab8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:40.403 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.402+0000 7ffab8ff9640 10 monclient(hunting): dump: 2024-03-25T18:20:40.403 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:40.404 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:40.404 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:40.404 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:40.404 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.404 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:40.404 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:40.404 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:40.404 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:40.404 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:40.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.403+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705325980 0 0) 0x7ffabc0aba50 con 0x7ffabc1430c0 2024-03-25T18:20:40.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.403+0000 7ffab8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:40.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.403+0000 7ffab8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:40.404 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.403+0000 7ffab8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:40.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.403+0000 7ffab8ff9640 10 monclient(hunting): my global_id is 4724 2024-03-25T18:20:40.405 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.403+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffabc137f50 con 0x7ffabc1430c0 2024-03-25T18:20:40.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.407+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 <== mon.0 v1:172.21.15.64:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 348410695 0 0) 0x7ffabc139830 con 0x7ffabc13f710 2024-03-25T18:20:40.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.407+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 >> v1:172.21.15.64:6790/0 conn(0x7ffabc1430c0 legacy=0x7ffabc136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.407+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 >> v1:172.21.15.86:6789/0 conn(0x7ffabc0a98f0 legacy=0x7ffabc134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.408+0000 7ffab8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:20:40.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.408+0000 7ffab8ff9640 1 monclient: found mon.a 2024-03-25T18:20:40.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.408+0000 7ffab8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:40.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.408+0000 7ffab8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:20:40.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.408+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffabc0abd30 con 0x7ffabc13f710 2024-03-25T18:20:40.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.408+0000 7ffab8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:20:40.409 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.408+0000 7ffab8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:20:40.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.408+0000 7ffab8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:40.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.408+0000 7ffac28ba640 5 monclient: authenticate success, global_id 4719 2024-03-25T18:20:40.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.408+0000 7ffac28ba640 10 monclient: _renew_subs 2024-03-25T18:20:40.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.408+0000 7ffac28ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:20:40.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.408+0000 7ffac28ba640 1 -- 172.21.15.64:0/1189310368 --> v1:172.21.15.64:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffabc137a00 con 0x7ffabc13f710 2024-03-25T18:20:40.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.408+0000 7ffac28ba640 10 client.4719.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:40.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.408+0000 7ffac28ba640 10 monclient: _renew_subs 2024-03-25T18:20:40.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.408+0000 7ffac28ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:20:40.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.408+0000 7ffac28ba640 1 -- 172.21.15.64:0/1189310368 --> v1:172.21.15.64:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffabc134970 con 0x7ffabc13f710 2024-03-25T18:20:40.410 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.408+0000 7ffac28ba640 1 librados: init done 2024-03-25T18:20:40.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.409+0000 7ffac28ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:40.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.409+0000 7ffac28ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:40.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.409+0000 7ffac28ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:20:40.411 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.409+0000 7ffac28ba640 1 -- 172.21.15.64:0/1189310368 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffabc0627b0 con 0x7ffabc13f710 2024-03-25T18:20:40.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.727+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 <== mon.0 v1:172.21.15.64:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffab0003610 con 0x7ffabc13f710 2024-03-25T18:20:40.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.727+0000 7ffab8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:40.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.727+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 <== mon.0 v1:172.21.15.64:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7ffab0001d80 con 0x7ffabc13f710 2024-03-25T18:20:40.728 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.727+0000 7ffab8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.727+0000 7ffab8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:40.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.727+0000 7ffab8ff9640 10 monclient: dump: 2024-03-25T18:20:40.729 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:40.729 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:40.729 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:40.729 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:40.729 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.729 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:40.729 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:40.729 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:40.729 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:40.729 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:40.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.727+0000 7ffaba7fc640 4 set_mon_vals no callback set 2024-03-25T18:20:40.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.728+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 <== mon.0 v1:172.21.15.64:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7ffab0019810 con 0x7ffabc13f710 2024-03-25T18:20:40.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.728+0000 7ffabbfff640 10 monclient: get_auth_request con 0x7ffaa806deb0 auth_method 0 2024-03-25T18:20:40.730 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.729+0000 7ffab8ff9640 10 client.4719.objecter ms_handle_connect 0x7ffaa806deb0 2024-03-25T18:20:40.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.734+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 <== mon.0 v1:172.21.15.64:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4227240091 0 0) 0x7ffab0053580 con 0x7ffabc13f710 2024-03-25T18:20:40.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.734+0000 7ffab8ff9640 10 client.4719.objecter ms_dispatch 0x7ffabc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:20:40.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.734+0000 7ffab8ff9640 3 client.4719.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:20:40.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.734+0000 7ffab8ff9640 3 client.4719.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:20:40.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.734+0000 7ffab8ff9640 20 client.4719.objecter dump_active .. 0 homeless 2024-03-25T18:20:40.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.734+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 <== mon.0 v1:172.21.15.64:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffab004eec0 con 0x7ffabc13f710 2024-03-25T18:20:40.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.734+0000 7ffab8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:40.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.734+0000 7ffab8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:40.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.978+0000 7ffac28ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:40.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.978+0000 7ffac28ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:40.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.978+0000 7ffac28ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.64:6789/0 2024-03-25T18:20:40.979 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:40.978+0000 7ffac28ba640 1 -- 172.21.15.64:0/1189310368 --> v1:172.21.15.64:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffabc0b71a0 con 0x7ffabc13f710 2024-03-25T18:20:41.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:41.007+0000 7ffab8ff9640 1 -- 172.21.15.64:0/1189310368 <== mon.0 v1:172.21.15.64:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 991200065) 0x7ffab0050fc0 con 0x7ffabc13f710 2024-03-25T18:20:41.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:41.007+0000 7ffab8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:41.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:41.007+0000 7ffab8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:41.008 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:20:41.008 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:19:38.421162+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:38.421162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:41.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:41.009+0000 7ffac28ba640 10 librados: watch_flush enter 2024-03-25T18:20:41.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:41.009+0000 7ffac28ba640 10 librados: watch_flush exit 2024-03-25T18:20:41.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:41.009+0000 7ffac28ba640 20 client.4719.objecter shutdown clearing up homeless session... 2024-03-25T18:20:41.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:41.009+0000 7ffac28ba640 10 client.4719.objecter successfully canceled tick 2024-03-25T18:20:41.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:41.009+0000 7ffac28ba640 1 -- 172.21.15.64:0/1189310368 >> v1:172.21.15.64:6800/2915543638 conn(0x7ffaa806deb0 legacy=0x7ffaa8070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:41.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:41.009+0000 7ffac28ba640 10 monclient: shutdown 2024-03-25T18:20:41.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:41.009+0000 7ffac28ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:41.011 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:41.009+0000 7ffac28ba640 1 -- 172.21.15.64:0/1189310368 >> v1:172.21.15.64:6789/0 conn(0x7ffabc13f710 legacy=0x7ffabc135b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:41.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:41.009+0000 7ffac28ba640 1 -- 172.21.15.64:0/1189310368 shutdown_connections 2024-03-25T18:20:41.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:41.010+0000 7ffac28ba640 1 -- 172.21.15.64:0/1189310368 >> 172.21.15.64:0/1189310368 conn(0x7ffabc0b7ea0 msgr2=0x7ffabc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:41.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:41.010+0000 7ffac28ba640 1 -- 172.21.15.64:0/1189310368 shutdown_connections 2024-03-25T18:20:41.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:41.010+0000 7ffac28ba640 1 -- 172.21.15.64:0/1189310368 wait complete. 2024-03-25T18:20:41.012 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:41.010+0000 7ffac28ba640 1 librados: shutdown 2024-03-25T18:20:41.022 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:42.374 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:25.486 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:42.379 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:25.491 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:42.387 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:11:25.466 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:42.390 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:11:25.468 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:42.402 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:25.515 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:42.406 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:25.518 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:42.408 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:11:25.486 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:42.412 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:25.524 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:42.494 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:25.606 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:43.687 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:26.799 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:43.693 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:11:26.772 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:43.696 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:26.809 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:43.738 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:11:26.817 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:43.742 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:26.854 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:43.749 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:11:26.828 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:43.750 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:26.863 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:43.771 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:26.884 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:43.794 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:26.907 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:47.829 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:47.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.947+0000 7f94c9b32640 10 monclient: get_monmap_and_config 2024-03-25T18:20:47.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.947+0000 7f94c9b32640 10 monclient: build_initial_monmap 2024-03-25T18:20:47.948 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.947+0000 7f94c9b32640 10 monclient: monmap: 2024-03-25T18:20:47.948 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:20:47.948 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:47.948 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:20:47.948353+0000 2024-03-25T18:20:47.948 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:20:47.948353+0000 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.948+0000 7f94c9b32640 1 Processor -- start 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.948+0000 7f94c9b32640 1 -- start start 2024-03-25T18:20:47.949 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.948+0000 7f94c9b32640 10 monclient: init 2024-03-25T18:20:47.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.949+0000 7f94c9b32640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:47.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.949+0000 7f94c9b32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:47.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.949+0000 7f94c9b32640 10 monclient(hunting): picked mon.noname-a con 0x7f94c40a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:20:47.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.949+0000 7f94c9b32640 10 monclient(hunting): picked mon.noname-c con 0x7f94c413f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:20:47.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.949+0000 7f94c9b32640 10 monclient(hunting): picked mon.noname-b con 0x7f94c41430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:20:47.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.949+0000 7f94c9b32640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c4142520 con 0x7f94c40a98f0 2024-03-25T18:20:47.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.949+0000 7f94c9b32640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c40b28a0 con 0x7f94c413f710 2024-03-25T18:20:47.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.949+0000 7f94c9b32640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c40b2a30 con 0x7f94c41430c0 2024-03-25T18:20:47.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.949+0000 7f94c9b32640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:47.951 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.949+0000 7f94c9b32640 10 monclient(hunting): authenticate will time out at 7634.187965s 2024-03-25T18:20:47.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.958+0000 7f94c3fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f94c41430c0 0x7f94c414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:53564/0 (socket says 172.21.15.64:53564) 2024-03-25T18:20:47.959 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.958+0000 7f94c3fff640 1 -- 172.21.15.64:0/698805293 learned_addr learned my addr 172.21.15.64:0/698805293 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:20:47.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.968+0000 7f94c2ffd640 1 -- 172.21.15.64:0/698805293 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136705024 0 0) 0x7f94c40b2a30 con 0x7f94c41430c0 2024-03-25T18:20:47.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.968+0000 7f94c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:47.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.968+0000 7f94c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:47.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.968+0000 7f94c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:47.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.968+0000 7f94c2ffd640 10 monclient(hunting): my global_id is 4727 2024-03-25T18:20:47.969 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.968+0000 7f94c2ffd640 1 -- 172.21.15.64:0/698805293 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94b0003520 con 0x7f94c41430c0 2024-03-25T18:20:47.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.980+0000 7f94c2ffd640 1 -- 172.21.15.64:0/698805293 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015474962 0 0) 0x7f94c4142520 con 0x7f94c40a98f0 2024-03-25T18:20:47.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.980+0000 7f94c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:47.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.980+0000 7f94c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:47.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.980+0000 7f94c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:47.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.980+0000 7f94c2ffd640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:20:47.981 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.980+0000 7f94c2ffd640 1 -- 172.21.15.64:0/698805293 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94c40b2a30 con 0x7f94c40a98f0 2024-03-25T18:20:47.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.984+0000 7f94c2ffd640 1 -- 172.21.15.64:0/698805293 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2537695709 0 0) 0x7f94b0003520 con 0x7f94c41430c0 2024-03-25T18:20:47.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.984+0000 7f94c2ffd640 1 -- 172.21.15.64:0/698805293 >> v1:172.21.15.86:6789/0 conn(0x7f94c413f710 legacy=0x7f94c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:47.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.984+0000 7f94c2ffd640 1 -- 172.21.15.64:0/698805293 >> v1:172.21.15.64:6789/0 conn(0x7f94c40a98f0 legacy=0x7f94c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:47.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.984+0000 7f94c2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:47.985 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.984+0000 7f94c2ffd640 1 monclient: found mon.noname-b 2024-03-25T18:20:47.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.984+0000 7f94c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:47.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.984+0000 7f94c2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:20:47.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.984+0000 7f94c2ffd640 1 -- 172.21.15.64:0/698805293 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c40b44c0 con 0x7f94c41430c0 2024-03-25T18:20:47.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.984+0000 7f94c2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:20:47.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.984+0000 7f94c2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:47.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.984+0000 7f94c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:47.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.984+0000 7f94c2ffd640 1 -- 172.21.15.64:0/698805293 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f94bc001690 con 0x7f94c41430c0 2024-03-25T18:20:47.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.984+0000 7f94c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:47.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.984+0000 7f94c2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:47.986 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.984+0000 7f94c2ffd640 10 monclient: dump: 2024-03-25T18:20:47.986 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:47.986 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:47.986 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:47.986 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:47.987 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:47.987 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:47.987 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:47.987 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:47.987 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:47.987 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:47.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.984+0000 7f94c9b32640 5 monclient: authenticate success, global_id 4727 2024-03-25T18:20:47.987 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:47.984+0000 7f94c9b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:48.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.022+0000 7f94c2ffd640 1 -- 172.21.15.64:0/698805293 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94bc0017f0 con 0x7f94c41430c0 2024-03-25T18:20:48.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.022+0000 7f94c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:48.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.022+0000 7f94c2ffd640 1 -- 172.21.15.64:0/698805293 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f94bc001c00 con 0x7f94c41430c0 2024-03-25T18:20:48.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.022+0000 7f94c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:48.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.022+0000 7f94c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:48.024 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.022+0000 7f94c2ffd640 10 monclient: dump: 2024-03-25T18:20:48.024 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:48.024 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:48.024 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:48.024 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:48.024 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:48.025 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:48.025 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:48.025 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:48.025 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:48.025 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:48.025 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.022+0000 7f94c9b32640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:48.025 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.022+0000 7f94c9b32640 4 set_mon_vals no callback set 2024-03-25T18:20:48.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.023+0000 7f94c9b32640 10 monclient: shutdown 2024-03-25T18:20:48.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.023+0000 7f94c9b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:48.027 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.023+0000 7f94c9b32640 1 -- 172.21.15.64:0/698805293 >> v1:172.21.15.64:6790/0 conn(0x7f94c41430c0 legacy=0x7f94c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.023+0000 7f94c9b32640 1 -- 172.21.15.64:0/698805293 shutdown_connections 2024-03-25T18:20:48.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.023+0000 7f94c9b32640 1 -- 172.21.15.64:0/698805293 >> 172.21.15.64:0/698805293 conn(0x7f94c40b7ea0 msgr2=0x7f94c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:48.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.023+0000 7f94c9b32640 1 -- 172.21.15.64:0/698805293 shutdown_connections 2024-03-25T18:20:48.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.023+0000 7f94c9b32640 1 -- 172.21.15.64:0/698805293 wait complete. 2024-03-25T18:20:48.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.024+0000 7f94c9b32640 10 monclient: build_initial_monmap 2024-03-25T18:20:48.028 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.024+0000 7f94c9b32640 10 monclient: monmap: 2024-03-25T18:20:48.028 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:20:48.028 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:48.028 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:20:48.028 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:20:48.028 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:48.028 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:20:48.028 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:20:48.028 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:20:48.028 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:20:48.029 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:48.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.024+0000 7f94c9b32640 1 librados: starting msgr at 2024-03-25T18:20:48.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.024+0000 7f94c9b32640 1 librados: starting objecter 2024-03-25T18:20:48.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.024+0000 7f94c9b32640 1 Processor -- start 2024-03-25T18:20:48.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.024+0000 7f94c9b32640 1 -- start start 2024-03-25T18:20:48.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.024+0000 7f94c9b32640 1 librados: setting wanted keys 2024-03-25T18:20:48.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.024+0000 7f94c9b32640 1 librados: calling monclient init 2024-03-25T18:20:48.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.024+0000 7f94c9b32640 10 monclient: init 2024-03-25T18:20:48.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.025+0000 7f94c9b32640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:48.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.025+0000 7f94c9b32640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:48.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.025+0000 7f94c9b32640 10 monclient(hunting): picked mon.noname-c con 0x7f94c40a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:20:48.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.025+0000 7f94c9b32640 10 monclient(hunting): picked mon.noname-b con 0x7f94c413f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:20:48.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.025+0000 7f94c9b32640 10 monclient(hunting): picked mon.noname-a con 0x7f94c41430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:20:48.029 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.025+0000 7f94c9b32640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c41daca0 con 0x7f94c41430c0 2024-03-25T18:20:48.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.025+0000 7f94c9b32640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c4138ab0 con 0x7f94c413f710 2024-03-25T18:20:48.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.025+0000 7f94c9b32640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c40aba90 con 0x7f94c40a98f0 2024-03-25T18:20:48.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.025+0000 7f94c9b32640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:48.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.025+0000 7f94c9b32640 10 monclient(hunting): authenticate will time out at 7634.263819s 2024-03-25T18:20:48.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.025+0000 7f94bbfff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f94c413f710 0x7f94c41db130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:33956/0 (socket says 172.21.15.64:33956) 2024-03-25T18:20:48.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.025+0000 7f94bbfff640 1 -- 172.21.15.64:0/648633764 learned_addr learned my addr 172.21.15.64:0/648633764 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:20:48.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.026+0000 7f94c0ff9640 10 client.?.objecter ms_handle_connect 0x7f94c413f710 2024-03-25T18:20:48.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.026+0000 7f94c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:48.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.026+0000 7f94c0ff9640 10 client.?.objecter ms_handle_connect 0x7f94c41430c0 2024-03-25T18:20:48.030 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.026+0000 7f94c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:48.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.093+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611862471 0 0) 0x7f94c41daca0 con 0x7f94c41430c0 2024-03-25T18:20:48.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.093+0000 7f94c0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:48.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.093+0000 7f94c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:48.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.093+0000 7f94c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:48.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.093+0000 7f94c0ff9640 10 monclient(hunting): my global_id is 4725 2024-03-25T18:20:48.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.093+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a4004eb0 con 0x7f94c41430c0 2024-03-25T18:20:48.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.105+0000 7f94c0ff9640 10 client.?.objecter ms_handle_connect 0x7f94c40a98f0 2024-03-25T18:20:48.106 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.105+0000 7f94c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:48.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.107+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2950692944 0 0) 0x7f94a4004eb0 con 0x7f94c41430c0 2024-03-25T18:20:48.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.107+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94c41daca0 con 0x7f94c41430c0 2024-03-25T18:20:48.108 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.107+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f94bc001b00 con 0x7f94c41430c0 2024-03-25T18:20:48.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.107+0000 7f94c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:48.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.107+0000 7f94c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:48.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.107+0000 7f94c0ff9640 10 monclient(hunting): dump: 2024-03-25T18:20:48.109 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:48.109 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:48.109 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:48.109 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:48.109 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:48.109 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:48.109 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:48.109 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:48.109 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:48.109 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:48.109 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.107+0000 7f94c0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:48.118 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.117+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993290669 0 0) 0x7f94c40aba90 con 0x7f94c40a98f0 2024-03-25T18:20:48.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.117+0000 7f94c0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:48.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.117+0000 7f94c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:48.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.117+0000 7f94c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:48.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.117+0000 7f94c0ff9640 10 monclient(hunting): my global_id is 4730 2024-03-25T18:20:48.119 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.117+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a4004eb0 con 0x7f94c40a98f0 2024-03-25T18:20:48.128 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.127+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3954582402 0 0) 0x7f94a4004eb0 con 0x7f94c40a98f0 2024-03-25T18:20:48.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.127+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94c40aba90 con 0x7f94c40a98f0 2024-03-25T18:20:48.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.127+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f94b4001d30 con 0x7f94c40a98f0 2024-03-25T18:20:48.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.127+0000 7f94c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:48.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.127+0000 7f94c0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:48.129 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.127+0000 7f94c0ff9640 10 monclient(hunting): dump: 2024-03-25T18:20:48.129 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:48.129 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:48.129 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:48.129 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:48.129 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:48.129 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:48.129 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:48.129 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:48.130 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:48.130 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:48.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.156+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494033128 0 0) 0x7f94c4138ab0 con 0x7f94c413f710 2024-03-25T18:20:48.157 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.156+0000 7f94c0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:48.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.156+0000 7f94c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:48.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.156+0000 7f94c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:48.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.156+0000 7f94c0ff9640 10 monclient(hunting): my global_id is 4735 2024-03-25T18:20:48.158 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.156+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a4004eb0 con 0x7f94c413f710 2024-03-25T18:20:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.157+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3933364045 0 0) 0x7f94c40aba90 con 0x7f94c40a98f0 2024-03-25T18:20:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.157+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 >> v1:172.21.15.86:6789/0 conn(0x7f94c413f710 legacy=0x7f94c41db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.157+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 >> v1:172.21.15.64:6789/0 conn(0x7f94c41430c0 legacy=0x7f94c41d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.157+0000 7f94c0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:20:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.157+0000 7f94c0ff9640 1 monclient: found mon.c 2024-03-25T18:20:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.157+0000 7f94c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.157+0000 7f94c0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:20:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.157+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c41d41e0 con 0x7f94c40a98f0 2024-03-25T18:20:48.159 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.157+0000 7f94c0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:20:48.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.157+0000 7f94c0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:20:48.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.157+0000 7f94c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:48.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.157+0000 7f94c9b32640 5 monclient: authenticate success, global_id 4730 2024-03-25T18:20:48.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.157+0000 7f94c9b32640 10 monclient: _renew_subs 2024-03-25T18:20:48.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.157+0000 7f94c9b32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:20:48.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.157+0000 7f94c9b32640 1 -- 172.21.15.64:0/648633764 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c41db840 con 0x7f94c40a98f0 2024-03-25T18:20:48.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.158+0000 7f94c9b32640 10 client.4730.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:48.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.158+0000 7f94c9b32640 10 monclient: _renew_subs 2024-03-25T18:20:48.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.158+0000 7f94c9b32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:20:48.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.158+0000 7f94c9b32640 1 -- 172.21.15.64:0/648633764 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c4138f00 con 0x7f94c40a98f0 2024-03-25T18:20:48.160 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.158+0000 7f94c9b32640 1 librados: init done 2024-03-25T18:20:48.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.159+0000 7f94c9b32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:48.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.159+0000 7f94c9b32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:48.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.159+0000 7f94c9b32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:20:48.161 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.159+0000 7f94c9b32640 1 -- 172.21.15.64:0/648633764 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c40627b0 con 0x7f94c40a98f0 2024-03-25T18:20:48.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.432+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94b4003990 con 0x7f94c40a98f0 2024-03-25T18:20:48.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.432+0000 7f94c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:48.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.432+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f94b4003ca0 con 0x7f94c40a98f0 2024-03-25T18:20:48.433 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.432+0000 7f94c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:48.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.432+0000 7f94c0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:48.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.432+0000 7f94c0ff9640 10 monclient: dump: 2024-03-25T18:20:48.434 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:48.434 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:48.434 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:48.434 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:48.434 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:48.434 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:48.434 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:48.434 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:48.434 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:48.434 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:48.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.432+0000 7f94c27fc640 4 set_mon_vals no callback set 2024-03-25T18:20:48.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.433+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f94b4019c70 con 0x7f94c40a98f0 2024-03-25T18:20:48.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.434+0000 7f94bbfff640 10 monclient: get_auth_request con 0x7f94a406ded0 auth_method 0 2024-03-25T18:20:48.435 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.434+0000 7f94c0ff9640 10 client.4730.objecter ms_handle_connect 0x7f94a406ded0 2024-03-25T18:20:48.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.440+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4227240091 0 0) 0x7f94b4053920 con 0x7f94c40a98f0 2024-03-25T18:20:48.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.440+0000 7f94c0ff9640 10 client.4730.objecter ms_dispatch 0x7f94c4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:20:48.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.440+0000 7f94c0ff9640 3 client.4730.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:20:48.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.440+0000 7f94c0ff9640 3 client.4730.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:20:48.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.440+0000 7f94c0ff9640 20 client.4730.objecter dump_active .. 0 homeless 2024-03-25T18:20:48.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.440+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f94b404f260 con 0x7f94c40a98f0 2024-03-25T18:20:48.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.440+0000 7f94c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:48.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.440+0000 7f94c0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:48.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.676+0000 7f94c9b32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:48.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.676+0000 7f94c9b32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:48.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.676+0000 7f94c9b32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:20:48.677 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.676+0000 7f94c9b32640 1 -- 172.21.15.64:0/648633764 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94c41db9d0 con 0x7f94c40a98f0 2024-03-25T18:20:48.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.713+0000 7f94c0ff9640 1 -- 172.21.15.64:0/648633764 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 991200065) 0x7f94b4051360 con 0x7f94c40a98f0 2024-03-25T18:20:48.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.713+0000 7f94c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:48.714 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.713+0000 7f94c0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:48.714 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:20:48.715 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:19:38.421162+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:38.421162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:48.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.715+0000 7f94c9b32640 10 librados: watch_flush enter 2024-03-25T18:20:48.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.715+0000 7f94c9b32640 10 librados: watch_flush exit 2024-03-25T18:20:48.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.715+0000 7f94c9b32640 20 client.4730.objecter shutdown clearing up homeless session... 2024-03-25T18:20:48.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.715+0000 7f94c9b32640 10 client.4730.objecter successfully canceled tick 2024-03-25T18:20:48.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.716+0000 7f94c9b32640 1 -- 172.21.15.64:0/648633764 >> v1:172.21.15.64:6800/2915543638 conn(0x7f94a406ded0 legacy=0x7f94a4070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.716+0000 7f94c9b32640 10 monclient: shutdown 2024-03-25T18:20:48.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.716+0000 7f94c9b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:48.717 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.716+0000 7f94c9b32640 1 -- 172.21.15.64:0/648633764 >> v1:172.21.15.64:6790/0 conn(0x7f94c40a98f0 legacy=0x7f94c41383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.716+0000 7f94c9b32640 1 -- 172.21.15.64:0/648633764 shutdown_connections 2024-03-25T18:20:48.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.716+0000 7f94c9b32640 1 -- 172.21.15.64:0/648633764 >> 172.21.15.64:0/648633764 conn(0x7f94c40b7ea0 msgr2=0x7f94c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:48.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.716+0000 7f94c9b32640 1 -- 172.21.15.64:0/648633764 shutdown_connections 2024-03-25T18:20:48.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.716+0000 7f94c9b32640 1 -- 172.21.15.64:0/648633764 wait complete. 2024-03-25T18:20:48.718 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:48.716+0000 7f94c9b32640 1 librados: shutdown 2024-03-25T18:20:48.729 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:55.536 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:55.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.651+0000 7f9aaff6d640 10 monclient: get_monmap_and_config 2024-03-25T18:20:55.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.651+0000 7f9aaff6d640 10 monclient: build_initial_monmap 2024-03-25T18:20:55.652 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.651+0000 7f9aaff6d640 10 monclient: monmap: 2024-03-25T18:20:55.652 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:20:55.652 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:55.652 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:20:55.651894+0000 2024-03-25T18:20:55.652 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:20:55.651894+0000 2024-03-25T18:20:55.652 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:55.652 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:55.652 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:20:55.652 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:20:55.652 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:20:55.653 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.651+0000 7f9aaff6d640 1 Processor -- start 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.651+0000 7f9aaff6d640 1 -- start start 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.651+0000 7f9aaff6d640 10 monclient: init 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.652+0000 7f9aaff6d640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.652+0000 7f9aaff6d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.652+0000 7f9aaff6d640 10 monclient(hunting): picked mon.noname-c con 0x7f9aa80a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.652+0000 7f9aaff6d640 10 monclient(hunting): picked mon.noname-b con 0x7f9aa813f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.652+0000 7f9aaff6d640 10 monclient(hunting): picked mon.noname-a con 0x7f9aa81430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.652+0000 7f9aaff6d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa8142520 con 0x7f9aa81430c0 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.652+0000 7f9aaff6d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa80b28a0 con 0x7f9aa80a98f0 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.652+0000 7f9aaff6d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa80b2a30 con 0x7f9aa813f710 2024-03-25T18:20:55.654 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.652+0000 7f9aaff6d640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:55.655 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.652+0000 7f9aaff6d640 10 monclient(hunting): authenticate will time out at 7641.891087s 2024-03-25T18:20:55.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.657+0000 7f9aadd09640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f9aa80a98f0 0x7f9aa80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:38614/0 (socket says 172.21.15.64:38614) 2024-03-25T18:20:55.658 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.657+0000 7f9aadd09640 1 -- 172.21.15.64:0/497248912 learned_addr learned my addr 172.21.15.64:0/497248912 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:20:55.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.666+0000 7f9aacd07640 1 -- 172.21.15.64:0/497248912 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625679185 0 0) 0x7f9aa8142520 con 0x7f9aa81430c0 2024-03-25T18:20:55.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.666+0000 7f9aacd07640 10 monclient(hunting): _negotiate 2024-03-25T18:20:55.667 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.666+0000 7f9aacd07640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:55.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.666+0000 7f9aacd07640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:55.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.666+0000 7f9aacd07640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:20:55.668 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.666+0000 7f9aacd07640 1 -- 172.21.15.64:0/497248912 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a94003520 con 0x7f9aa81430c0 2024-03-25T18:20:55.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.672+0000 7f9aacd07640 1 -- 172.21.15.64:0/497248912 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129236200 0 0) 0x7f9aa80b2a30 con 0x7f9aa813f710 2024-03-25T18:20:55.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.672+0000 7f9aacd07640 10 monclient(hunting): _negotiate 2024-03-25T18:20:55.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.672+0000 7f9aacd07640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:55.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.672+0000 7f9aacd07640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:55.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.672+0000 7f9aacd07640 10 monclient(hunting): my global_id is 4733 2024-03-25T18:20:55.673 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.672+0000 7f9aacd07640 1 -- 172.21.15.64:0/497248912 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa8142520 con 0x7f9aa813f710 2024-03-25T18:20:55.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.673+0000 7f9aacd07640 1 -- 172.21.15.64:0/497248912 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3640878104 0 0) 0x7f9aa80b28a0 con 0x7f9aa80a98f0 2024-03-25T18:20:55.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.673+0000 7f9aacd07640 10 monclient(hunting): _negotiate 2024-03-25T18:20:55.674 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.673+0000 7f9aacd07640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:55.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.673+0000 7f9aacd07640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:55.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.673+0000 7f9aacd07640 10 monclient(hunting): my global_id is 4738 2024-03-25T18:20:55.675 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.673+0000 7f9aacd07640 1 -- 172.21.15.64:0/497248912 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa80b2a30 con 0x7f9aa80a98f0 2024-03-25T18:20:55.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.685+0000 7f9aacd07640 1 -- 172.21.15.64:0/497248912 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3709505253 0 0) 0x7f9a94003520 con 0x7f9aa81430c0 2024-03-25T18:20:55.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.685+0000 7f9aacd07640 1 -- 172.21.15.64:0/497248912 >> v1:172.21.15.64:6790/0 conn(0x7f9aa813f710 legacy=0x7f9aa81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.685+0000 7f9aacd07640 1 -- 172.21.15.64:0/497248912 >> v1:172.21.15.86:6789/0 conn(0x7f9aa80a98f0 legacy=0x7f9aa80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.685+0000 7f9aacd07640 10 monclient: _finish_hunting 0 2024-03-25T18:20:55.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.685+0000 7f9aacd07640 1 monclient: found mon.noname-a 2024-03-25T18:20:55.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.685+0000 7f9aacd07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:55.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.685+0000 7f9aacd07640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.64:6789/0 2024-03-25T18:20:55.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.685+0000 7f9aacd07640 1 -- 172.21.15.64:0/497248912 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa80b44c0 con 0x7f9aa81430c0 2024-03-25T18:20:55.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.685+0000 7f9aacd07640 10 monclient: _finish_auth 0 2024-03-25T18:20:55.686 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.685+0000 7f9aacd07640 10 monclient: _check_auth_tickets 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.685+0000 7f9aacd07640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.685+0000 7f9aaff6d640 5 monclient: authenticate success, global_id 4728 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.685+0000 7f9aacd07640 1 -- 172.21.15.64:0/497248912 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9aa0001a10 con 0x7f9aa81430c0 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.685+0000 7f9aacd07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.685+0000 7f9aacd07640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.685+0000 7f9aacd07640 10 monclient: dump: 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:55.687 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:55.688 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:55.688 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.685+0000 7f9aaff6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.707+0000 7f9aacd07640 1 -- 172.21.15.64:0/497248912 <== mon.0 v1:172.21.15.64:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9aa0001b70 con 0x7f9aa81430c0 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.707+0000 7f9aacd07640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.707+0000 7f9aacd07640 1 -- 172.21.15.64:0/497248912 <== mon.0 v1:172.21.15.64:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9aa0001f80 con 0x7f9aa81430c0 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.707+0000 7f9aaff6d640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.707+0000 7f9aaff6d640 4 set_mon_vals no callback set 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.707+0000 7f9aacd07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.707+0000 7f9aacd07640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.707+0000 7f9aacd07640 10 monclient: dump: 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:55.708 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:55.709 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:55.709 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:55.709 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:55.709 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:55.709 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:55.709 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:55.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.707+0000 7f9aaff6d640 10 monclient: shutdown 2024-03-25T18:20:55.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.707+0000 7f9aaff6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:55.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.707+0000 7f9aaff6d640 1 -- 172.21.15.64:0/497248912 >> v1:172.21.15.64:6789/0 conn(0x7f9aa81430c0 legacy=0x7f9aa814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.708+0000 7f9aaff6d640 1 -- 172.21.15.64:0/497248912 shutdown_connections 2024-03-25T18:20:55.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.708+0000 7f9aaff6d640 1 -- 172.21.15.64:0/497248912 >> 172.21.15.64:0/497248912 conn(0x7f9aa80b7ea0 msgr2=0x7f9aa80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:55.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.708+0000 7f9aaff6d640 1 -- 172.21.15.64:0/497248912 shutdown_connections 2024-03-25T18:20:55.710 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.708+0000 7f9aaff6d640 1 -- 172.21.15.64:0/497248912 wait complete. 2024-03-25T18:20:55.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.708+0000 7f9aaff6d640 10 monclient: build_initial_monmap 2024-03-25T18:20:55.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.708+0000 7f9aaff6d640 10 monclient: monmap: 2024-03-25T18:20:55.711 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:20:55.711 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:55.711 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:20:55.711 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:20:55.711 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:55.711 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:20:55.711 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:20:55.711 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:20:55.711 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:20:55.711 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:55.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.709+0000 7f9aaff6d640 1 librados: starting msgr at 2024-03-25T18:20:55.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.709+0000 7f9aaff6d640 1 librados: starting objecter 2024-03-25T18:20:55.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.709+0000 7f9aaff6d640 1 Processor -- start 2024-03-25T18:20:55.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.709+0000 7f9aaff6d640 1 -- start start 2024-03-25T18:20:55.711 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.709+0000 7f9aaff6d640 1 librados: setting wanted keys 2024-03-25T18:20:55.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.709+0000 7f9aaff6d640 1 librados: calling monclient init 2024-03-25T18:20:55.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.709+0000 7f9aaff6d640 10 monclient: init 2024-03-25T18:20:55.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.710+0000 7f9aaff6d640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:55.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.710+0000 7f9aaff6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:55.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.710+0000 7f9aaff6d640 10 monclient(hunting): picked mon.noname-b con 0x7f9aa80a98f0 addr v1:172.21.15.86:6789/0 2024-03-25T18:20:55.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.710+0000 7f9aaff6d640 10 monclient(hunting): picked mon.noname-a con 0x7f9aa813f710 addr v1:172.21.15.64:6789/0 2024-03-25T18:20:55.712 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.710+0000 7f9aaff6d640 10 monclient(hunting): picked mon.noname-c con 0x7f9aa81430c0 addr v1:172.21.15.64:6790/0 2024-03-25T18:20:55.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.710+0000 7f9aaff6d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa8136f00 con 0x7f9aa813f710 2024-03-25T18:20:55.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.710+0000 7f9aaff6d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa8136c20 con 0x7f9aa80a98f0 2024-03-25T18:20:55.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.710+0000 7f9aaff6d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa81339d0 con 0x7f9aa81430c0 2024-03-25T18:20:55.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.710+0000 7f9aaff6d640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:55.713 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.710+0000 7f9aaff6d640 10 monclient(hunting): authenticate will time out at 7641.948711s 2024-03-25T18:20:55.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.718+0000 7f9aae50a640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f9aa81430c0 0x7f9aa81332c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:58194/0 (socket says 172.21.15.64:58194) 2024-03-25T18:20:55.719 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.718+0000 7f9aae50a640 1 -- 172.21.15.64:0/1207834626 learned_addr learned my addr 172.21.15.64:0/1207834626 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:20:55.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.720+0000 7f9a9e7fc640 10 client.?.objecter ms_handle_connect 0x7f9aa81430c0 2024-03-25T18:20:55.721 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.720+0000 7f9a9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:55.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.724+0000 7f9a9e7fc640 10 client.?.objecter ms_handle_connect 0x7f9aa813f710 2024-03-25T18:20:55.724 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.724+0000 7f9a9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:55.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.728+0000 7f9a9e7fc640 10 client.?.objecter ms_handle_connect 0x7f9aa80a98f0 2024-03-25T18:20:55.729 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.728+0000 7f9a9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:55.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.732+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945660787 0 0) 0x7f9aa8136f00 con 0x7f9aa813f710 2024-03-25T18:20:55.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.732+0000 7f9a9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:55.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.732+0000 7f9a9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:55.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.732+0000 7f9a9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:55.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.732+0000 7f9a9e7fc640 10 monclient(hunting): my global_id is 4731 2024-03-25T18:20:55.733 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.732+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a8c004eb0 con 0x7f9aa813f710 2024-03-25T18:20:55.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.734+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3493189398 0 0) 0x7f9aa81339d0 con 0x7f9aa81430c0 2024-03-25T18:20:55.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.734+0000 7f9a9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:55.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.734+0000 7f9a9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:55.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.734+0000 7f9a9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:55.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.734+0000 7f9a9e7fc640 10 monclient(hunting): my global_id is 4736 2024-03-25T18:20:55.735 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.734+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa8136f00 con 0x7f9aa81430c0 2024-03-25T18:20:55.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.738+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480158956 0 0) 0x7f9aa8136c20 con 0x7f9aa80a98f0 2024-03-25T18:20:55.739 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.738+0000 7f9a9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:55.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.738+0000 7f9a9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:55.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.738+0000 7f9a9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:55.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.738+0000 7f9a9e7fc640 10 monclient(hunting): my global_id is 4741 2024-03-25T18:20:55.740 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.738+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa81339d0 con 0x7f9aa80a98f0 2024-03-25T18:20:55.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.751+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2091801477 0 0) 0x7f9aa8136f00 con 0x7f9aa81430c0 2024-03-25T18:20:55.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.751+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9aa8136c20 con 0x7f9aa81430c0 2024-03-25T18:20:55.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.751+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9aa0001e80 con 0x7f9aa81430c0 2024-03-25T18:20:55.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.751+0000 7f9a9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:55.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.751+0000 7f9a9e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:55.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.751+0000 7f9a9e7fc640 10 monclient(hunting): dump: 2024-03-25T18:20:55.752 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:55.752 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:55.752 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:55.752 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:55.752 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:55.752 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:55.753 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:55.753 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:55.753 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:55.753 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:55.753 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.751+0000 7f9a9e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:55.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.757+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1558413562 0 0) 0x7f9aa81339d0 con 0x7f9aa80a98f0 2024-03-25T18:20:55.758 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.757+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9aa8136f00 con 0x7f9aa80a98f0 2024-03-25T18:20:55.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.757+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9aa4001830 con 0x7f9aa80a98f0 2024-03-25T18:20:55.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.757+0000 7f9a9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:55.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.757+0000 7f9a9e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:55.759 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.757+0000 7f9a9e7fc640 10 monclient(hunting): dump: 2024-03-25T18:20:55.759 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:55.759 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:55.759 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:55.759 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:55.759 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:55.759 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:55.759 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:55.759 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:55.759 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:55.759 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:55.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.758+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 271851867 0 0) 0x7f9a8c004eb0 con 0x7f9aa813f710 2024-03-25T18:20:55.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.758+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9aa81339d0 con 0x7f9aa813f710 2024-03-25T18:20:55.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.758+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9a98001970 con 0x7f9aa813f710 2024-03-25T18:20:55.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.758+0000 7f9a9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:55.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.758+0000 7f9a9e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:55.760 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.758+0000 7f9a9e7fc640 10 monclient(hunting): dump: 2024-03-25T18:20:55.760 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:55.760 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:55.761 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:55.761 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:55.761 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:55.761 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:55.761 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:55.761 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:55.761 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:55.761 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:55.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2348972406 0 0) 0x7f9aa8136c20 con 0x7f9aa81430c0 2024-03-25T18:20:55.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 >> v1:172.21.15.86:6789/0 conn(0x7f9aa80a98f0 legacy=0x7f9aa8137470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 >> v1:172.21.15.64:6789/0 conn(0x7f9aa813f710 legacy=0x7f9aa8132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9a9e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:55.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9a9e7fc640 1 monclient: found mon.c 2024-03-25T18:20:55.769 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9a9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:55.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9a9e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:20:55.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa8137b80 con 0x7f9aa81430c0 2024-03-25T18:20:55.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9a9e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:55.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9a9e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:55.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9a9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:55.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9aaff6d640 5 monclient: authenticate success, global_id 4736 2024-03-25T18:20:55.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9aaff6d640 10 monclient: _renew_subs 2024-03-25T18:20:55.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9aaff6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:20:55.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9aaff6d640 1 -- 172.21.15.64:0/1207834626 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa8139750 con 0x7f9aa81430c0 2024-03-25T18:20:55.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9aaff6d640 10 client.4736.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:55.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9aaff6d640 10 monclient: _renew_subs 2024-03-25T18:20:55.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9aaff6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:20:55.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9aaff6d640 1 -- 172.21.15.64:0/1207834626 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa81dab70 con 0x7f9aa81430c0 2024-03-25T18:20:55.770 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.768+0000 7f9aaff6d640 1 librados: init done 2024-03-25T18:20:55.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.769+0000 7f9aaff6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:55.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.769+0000 7f9aaff6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:55.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.769+0000 7f9aaff6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:20:55.771 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:55.769+0000 7f9aaff6d640 1 -- 172.21.15.64:0/1207834626 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa80627b0 con 0x7f9aa81430c0 2024-03-25T18:20:56.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.085+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9aa0001590 con 0x7f9aa81430c0 2024-03-25T18:20:56.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.085+0000 7f9a9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:56.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.085+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9aa0003dc0 con 0x7f9aa81430c0 2024-03-25T18:20:56.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.085+0000 7f9a9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:56.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.085+0000 7f9a9e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:56.086 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.085+0000 7f9a9e7fc640 10 monclient: dump: 2024-03-25T18:20:56.086 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:20:56.086 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:20:56.086 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:56.086 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:20:56.086 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:56.087 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:20:56.087 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:20:56.087 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:20:56.087 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:20:56.087 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:20:56.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.085+0000 7f9a9ffff640 4 set_mon_vals no callback set 2024-03-25T18:20:56.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.086+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f9aa0003fe0 con 0x7f9aa81430c0 2024-03-25T18:20:56.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.086+0000 7f9aadd09640 10 monclient: get_auth_request con 0x7f9a8c06e0f0 auth_method 0 2024-03-25T18:20:56.087 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.087+0000 7f9a9e7fc640 10 client.4736.objecter ms_handle_connect 0x7f9a8c06e0f0 2024-03-25T18:20:56.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.093+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4227240091 0 0) 0x7f9aa0054aa0 con 0x7f9aa81430c0 2024-03-25T18:20:56.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.093+0000 7f9a9e7fc640 10 client.4736.objecter ms_dispatch 0x7f9aa8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:20:56.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.093+0000 7f9a9e7fc640 3 client.4736.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:20:56.094 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.093+0000 7f9a9e7fc640 3 client.4736.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:20:56.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.093+0000 7f9a9e7fc640 20 client.4736.objecter dump_active .. 0 homeless 2024-03-25T18:20:56.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.093+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9aa00505b0 con 0x7f9aa81430c0 2024-03-25T18:20:56.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.093+0000 7f9a9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:56.095 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.093+0000 7f9a9e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:56.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.315+0000 7f9aaff6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:56.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.315+0000 7f9aaff6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:56.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.315+0000 7f9aaff6d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:20:56.316 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.315+0000 7f9aaff6d640 1 -- 172.21.15.64:0/1207834626 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9aa8137d10 con 0x7f9aa81430c0 2024-03-25T18:20:56.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.412+0000 7f9a9e7fc640 1 -- 172.21.15.64:0/1207834626 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 991200065) 0x7f9aa00524e0 con 0x7f9aa81430c0 2024-03-25T18:20:56.413 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.412+0000 7f9a9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:56.414 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.412+0000 7f9a9e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:56.414 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:20:56.414 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:19:38.421162+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:38.421162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:56.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.414+0000 7f9aaff6d640 10 librados: watch_flush enter 2024-03-25T18:20:56.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.414+0000 7f9aaff6d640 10 librados: watch_flush exit 2024-03-25T18:20:56.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.414+0000 7f9aaff6d640 20 client.4736.objecter shutdown clearing up homeless session... 2024-03-25T18:20:56.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.414+0000 7f9aaff6d640 10 client.4736.objecter successfully canceled tick 2024-03-25T18:20:56.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.415+0000 7f9aaff6d640 1 -- 172.21.15.64:0/1207834626 >> v1:172.21.15.64:6800/2915543638 conn(0x7f9a8c06e0f0 legacy=0x7f9a8c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.415+0000 7f9aaff6d640 10 monclient: shutdown 2024-03-25T18:20:56.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.415+0000 7f9aaff6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:56.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.415+0000 7f9aaff6d640 1 -- 172.21.15.64:0/1207834626 >> v1:172.21.15.64:6790/0 conn(0x7f9aa81430c0 legacy=0x7f9aa81332c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.415+0000 7f9aaff6d640 1 -- 172.21.15.64:0/1207834626 shutdown_connections 2024-03-25T18:20:56.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.415+0000 7f9aaff6d640 1 -- 172.21.15.64:0/1207834626 >> 172.21.15.64:0/1207834626 conn(0x7f9aa80b7ea0 msgr2=0x7f9aa813eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:56.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.415+0000 7f9aaff6d640 1 -- 172.21.15.64:0/1207834626 shutdown_connections 2024-03-25T18:20:56.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.415+0000 7f9aaff6d640 1 -- 172.21.15.64:0/1207834626 wait complete. 2024-03-25T18:20:56.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:20:56.415+0000 7f9aaff6d640 1 librados: shutdown 2024-03-25T18:20:56.428 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:03.234 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:03.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.346+0000 7f983ad8d640 10 monclient: get_monmap_and_config 2024-03-25T18:21:03.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.346+0000 7f983ad8d640 10 monclient: build_initial_monmap 2024-03-25T18:21:03.348 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.346+0000 7f983ad8d640 10 monclient: monmap: 2024-03-25T18:21:03.348 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:21:03.348 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:21:03.348 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:21:03.348140+0000 2024-03-25T18:21:03.348 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:21:03.348140+0000 2024-03-25T18:21:03.348 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:03.348 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:21:03.348 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:21:03.348 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-25T18:21:03.348 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-b 2024-03-25T18:21:03.349 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:21:03.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.347+0000 7f983ad8d640 1 Processor -- start 2024-03-25T18:21:03.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.347+0000 7f983ad8d640 1 -- start start 2024-03-25T18:21:03.349 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.347+0000 7f983ad8d640 10 monclient: init 2024-03-25T18:21:03.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.348+0000 7f983ad8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:03.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.348+0000 7f983ad8d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:03.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.348+0000 7f983ad8d640 10 monclient(hunting): picked mon.noname-a con 0x7f98340a98f0 addr v1:172.21.15.64:6789/0 2024-03-25T18:21:03.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.348+0000 7f983ad8d640 10 monclient(hunting): picked mon.noname-b con 0x7f983413f710 addr v1:172.21.15.64:6790/0 2024-03-25T18:21:03.350 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.348+0000 7f983ad8d640 10 monclient(hunting): picked mon.noname-c con 0x7f98341430c0 addr v1:172.21.15.86:6789/0 2024-03-25T18:21:03.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.348+0000 7f983ad8d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9834142520 con 0x7f98340a98f0 2024-03-25T18:21:03.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.348+0000 7f983ad8d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98340b28a0 con 0x7f98341430c0 2024-03-25T18:21:03.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.348+0000 7f983ad8d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98340b2a30 con 0x7f983413f710 2024-03-25T18:21:03.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.348+0000 7f983ad8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:03.351 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.348+0000 7f983ad8d640 10 monclient(hunting): authenticate will time out at 7649.587765s 2024-03-25T18:21:03.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.355+0000 7f9833fff640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f983413f710 0x7f98341429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:37722/0 (socket says 172.21.15.64:37722) 2024-03-25T18:21:03.357 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.355+0000 7f9833fff640 1 -- 172.21.15.64:0/3802592130 learned_addr learned my addr 172.21.15.64:0/3802592130 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:21:03.370 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.368+0000 7f98337fe640 1 -- 172.21.15.64:0/3802592130 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097994835 0 0) 0x7f98340b2a30 con 0x7f983413f710 2024-03-25T18:21:03.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.369+0000 7f98337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:03.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.369+0000 7f98337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:03.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.369+0000 7f98337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:03.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.369+0000 7f98337fe640 10 monclient(hunting): my global_id is 4739 2024-03-25T18:21:03.371 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.369+0000 7f98337fe640 1 -- 172.21.15.64:0/3802592130 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9824003520 con 0x7f983413f710 2024-03-25T18:21:03.380 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.378+0000 7f98337fe640 1 -- 172.21.15.64:0/3802592130 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 705892216 0 0) 0x7f9824003520 con 0x7f983413f710 2024-03-25T18:21:03.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.378+0000 7f98337fe640 1 -- 172.21.15.64:0/3802592130 >> v1:172.21.15.86:6789/0 conn(0x7f98341430c0 legacy=0x7f983414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.378+0000 7f98337fe640 1 -- 172.21.15.64:0/3802592130 >> v1:172.21.15.64:6789/0 conn(0x7f98340a98f0 legacy=0x7f98340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.378+0000 7f98337fe640 10 monclient: _finish_hunting 0 2024-03-25T18:21:03.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.378+0000 7f98337fe640 1 monclient: found mon.noname-b 2024-03-25T18:21:03.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.378+0000 7f98337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:03.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.378+0000 7f98337fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.64:6790/0 2024-03-25T18:21:03.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.378+0000 7f98337fe640 1 -- 172.21.15.64:0/3802592130 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98340b44c0 con 0x7f983413f710 2024-03-25T18:21:03.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.378+0000 7f98337fe640 10 monclient: _finish_auth 0 2024-03-25T18:21:03.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.378+0000 7f98337fe640 10 monclient: _check_auth_tickets 2024-03-25T18:21:03.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.378+0000 7f98337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:03.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.378+0000 7f98337fe640 1 -- 172.21.15.64:0/3802592130 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9820001690 con 0x7f983413f710 2024-03-25T18:21:03.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.378+0000 7f98337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.381 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.378+0000 7f98337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:03.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.378+0000 7f98337fe640 10 monclient: dump: 2024-03-25T18:21:03.382 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:21:03.382 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:21:03.382 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:21:03.382 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:21:03.382 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.382 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:21:03.382 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:21:03.382 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:21:03.382 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:21:03.382 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:21:03.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.378+0000 7f983ad8d640 5 monclient: authenticate success, global_id 4739 2024-03-25T18:21:03.382 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.378+0000 7f983ad8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:03.393 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.391+0000 7f98337fe640 1 -- 172.21.15.64:0/3802592130 <== mon.2 v1:172.21.15.64:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98200017f0 con 0x7f983413f710 2024-03-25T18:21:03.416 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.391+0000 7f98337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:03.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.391+0000 7f98337fe640 1 -- 172.21.15.64:0/3802592130 <== mon.2 v1:172.21.15.64:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9820001c40 con 0x7f983413f710 2024-03-25T18:21:03.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.391+0000 7f983ad8d640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:03.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.391+0000 7f983ad8d640 4 set_mon_vals no callback set 2024-03-25T18:21:03.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.391+0000 7f98337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.391+0000 7f98337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:03.417 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.391+0000 7f98337fe640 10 monclient: dump: 2024-03-25T18:21:03.417 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:21:03.417 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:21:03.417 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:21:03.417 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:21:03.417 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.417 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:21:03.417 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:21:03.417 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:21:03.417 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:21:03.418 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:21:03.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.392+0000 7f983ad8d640 10 monclient: shutdown 2024-03-25T18:21:03.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.392+0000 7f983ad8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:03.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.392+0000 7f983ad8d640 1 -- 172.21.15.64:0/3802592130 >> v1:172.21.15.64:6790/0 conn(0x7f983413f710 legacy=0x7f98341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.392+0000 7f983ad8d640 1 -- 172.21.15.64:0/3802592130 shutdown_connections 2024-03-25T18:21:03.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.392+0000 7f983ad8d640 1 -- 172.21.15.64:0/3802592130 >> 172.21.15.64:0/3802592130 conn(0x7f98340b7ea0 msgr2=0x7f98340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:03.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.392+0000 7f983ad8d640 1 -- 172.21.15.64:0/3802592130 shutdown_connections 2024-03-25T18:21:03.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.392+0000 7f983ad8d640 1 -- 172.21.15.64:0/3802592130 wait complete. 2024-03-25T18:21:03.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.392+0000 7f983ad8d640 10 monclient: build_initial_monmap 2024-03-25T18:21:03.421 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.392+0000 7f983ad8d640 10 monclient: monmap: 2024-03-25T18:21:03.422 INFO:teuthology.misc.health.smithi064.stderr:epoch 0 2024-03-25T18:21:03.422 INFO:teuthology.misc.health.smithi064.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:03.422 INFO:teuthology.misc.health.smithi064.stderr:last_changed 0.000000 2024-03-25T18:21:03.422 INFO:teuthology.misc.health.smithi064.stderr:created 0.000000 2024-03-25T18:21:03.422 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:03.422 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 1 2024-03-25T18:21:03.422 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.noname-a 2024-03-25T18:21:03.422 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-25T18:21:03.422 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.noname-c 2024-03-25T18:21:03.422 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:21:03.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.393+0000 7f983ad8d640 1 librados: starting msgr at 2024-03-25T18:21:03.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.393+0000 7f983ad8d640 1 librados: starting objecter 2024-03-25T18:21:03.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.393+0000 7f983ad8d640 1 Processor -- start 2024-03-25T18:21:03.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.393+0000 7f983ad8d640 1 -- start start 2024-03-25T18:21:03.422 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.393+0000 7f983ad8d640 1 librados: setting wanted keys 2024-03-25T18:21:03.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.393+0000 7f983ad8d640 1 librados: calling monclient init 2024-03-25T18:21:03.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.393+0000 7f983ad8d640 10 monclient: init 2024-03-25T18:21:03.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.394+0000 7f983ad8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:03.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.394+0000 7f983ad8d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:03.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.394+0000 7f983ad8d640 10 monclient(hunting): picked mon.noname-c con 0x7f98340a98f0 addr v1:172.21.15.64:6790/0 2024-03-25T18:21:03.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.394+0000 7f983ad8d640 10 monclient(hunting): picked mon.noname-b con 0x7f983413f710 addr v1:172.21.15.86:6789/0 2024-03-25T18:21:03.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.394+0000 7f983ad8d640 10 monclient(hunting): picked mon.noname-a con 0x7f98341430c0 addr v1:172.21.15.64:6789/0 2024-03-25T18:21:03.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.394+0000 7f983ad8d640 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98341dba50 con 0x7f98341430c0 2024-03-25T18:21:03.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.394+0000 7f983ad8d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98340704c0 con 0x7f983413f710 2024-03-25T18:21:03.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.394+0000 7f983ad8d640 1 -- --> v1:172.21.15.64:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f983406dea0 con 0x7f98340a98f0 2024-03-25T18:21:03.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.394+0000 7f983ad8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:03.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.394+0000 7f983ad8d640 10 monclient(hunting): authenticate will time out at 7649.633970s 2024-03-25T18:21:03.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.404+0000 7f9838b29640 1 --1- >> v1:172.21.15.64:6790/0 conn(0x7f98340a98f0 0x7f98341dbee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.64:6790/0 says I am v1:172.21.15.64:37750/0 (socket says 172.21.15.64:37750) 2024-03-25T18:21:03.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.404+0000 7f9838b29640 1 -- 172.21.15.64:0/640232983 learned_addr learned my addr 172.21.15.64:0/640232983 (peer_addr_for_me v1:172.21.15.64:0/0) 2024-03-25T18:21:03.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.406+0000 7f98317fa640 10 client.?.objecter ms_handle_connect 0x7f98340a98f0 2024-03-25T18:21:03.423 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.406+0000 7f98317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:03.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.406+0000 7f98317fa640 10 client.?.objecter ms_handle_connect 0x7f98341430c0 2024-03-25T18:21:03.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.406+0000 7f98317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:03.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.407+0000 7f98317fa640 10 client.?.objecter ms_handle_connect 0x7f983413f710 2024-03-25T18:21:03.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.407+0000 7f98317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:03.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.412+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 <== mon.2 v1:172.21.15.64:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460327565 0 0) 0x7f983406dea0 con 0x7f98340a98f0 2024-03-25T18:21:03.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.412+0000 7f98317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:03.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.412+0000 7f98317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:03.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.412+0000 7f98317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:03.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.412+0000 7f98317fa640 10 monclient(hunting): my global_id is 4742 2024-03-25T18:21:03.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.412+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 --> v1:172.21.15.64:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9818004eb0 con 0x7f98340a98f0 2024-03-25T18:21:03.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.416+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880720556 0 0) 0x7f98341dba50 con 0x7f98341430c0 2024-03-25T18:21:03.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.416+0000 7f98317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:03.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.416+0000 7f98317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:03.424 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.416+0000 7f98317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:03.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.416+0000 7f98317fa640 10 monclient(hunting): my global_id is 4737 2024-03-25T18:21:03.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.416+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f983406dea0 con 0x7f98341430c0 2024-03-25T18:21:03.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.420+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842631080 0 0) 0x7f98340704c0 con 0x7f983413f710 2024-03-25T18:21:03.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.420+0000 7f98317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:03.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.420+0000 7f98317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:03.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.420+0000 7f98317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:03.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.420+0000 7f98317fa640 10 monclient(hunting): my global_id is 4744 2024-03-25T18:21:03.425 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.420+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98341dba50 con 0x7f983413f710 2024-03-25T18:21:03.429 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.427+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 <== mon.2 v1:172.21.15.64:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4077686802 0 0) 0x7f9818004eb0 con 0x7f98340a98f0 2024-03-25T18:21:03.429 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.427+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 --> v1:172.21.15.64:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98340704c0 con 0x7f98340a98f0 2024-03-25T18:21:03.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.427+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 <== mon.2 v1:172.21.15.64:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f98280019d0 con 0x7f98340a98f0 2024-03-25T18:21:03.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.427+0000 7f98317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.427+0000 7f98317fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:03.430 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.427+0000 7f98317fa640 10 monclient(hunting): dump: 2024-03-25T18:21:03.430 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:21:03.430 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:21:03.430 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:21:03.430 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:21:03.430 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.430 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:21:03.430 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:21:03.430 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:21:03.430 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:21:03.430 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:21:03.431 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.427+0000 7f98317fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:03.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.432+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2849078942 0 0) 0x7f983406dea0 con 0x7f98341430c0 2024-03-25T18:21:03.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.432+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 --> v1:172.21.15.64:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9818004eb0 con 0x7f98341430c0 2024-03-25T18:21:03.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.432+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f982c001b10 con 0x7f98341430c0 2024-03-25T18:21:03.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.432+0000 7f98317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.432+0000 7f98317fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:03.434 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.432+0000 7f98317fa640 10 monclient(hunting): dump: 2024-03-25T18:21:03.435 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:21:03.435 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:21:03.435 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:21:03.435 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:21:03.435 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.435 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:21:03.435 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:21:03.435 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:21:03.435 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:21:03.435 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:21:03.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.436+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 <== mon.2 v1:172.21.15.64:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 330301055 0 0) 0x7f98340704c0 con 0x7f98340a98f0 2024-03-25T18:21:03.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.436+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 >> v1:172.21.15.86:6789/0 conn(0x7f983413f710 legacy=0x7f983406e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.436+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 >> v1:172.21.15.64:6789/0 conn(0x7f98341430c0 legacy=0x7f98341e6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.436+0000 7f98317fa640 10 monclient: _finish_hunting 0 2024-03-25T18:21:03.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.436+0000 7f98317fa640 1 monclient: found mon.c 2024-03-25T18:21:03.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.436+0000 7f98317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:03.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.437+0000 7f98317fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:21:03.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.437+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 --> v1:172.21.15.64:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98341dc5f0 con 0x7f98340a98f0 2024-03-25T18:21:03.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.437+0000 7f98317fa640 10 monclient: _finish_auth 0 2024-03-25T18:21:03.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.437+0000 7f98317fa640 10 monclient: _check_auth_tickets 2024-03-25T18:21:03.440 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.437+0000 7f98317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:03.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.437+0000 7f983ad8d640 5 monclient: authenticate success, global_id 4742 2024-03-25T18:21:03.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.437+0000 7f983ad8d640 10 monclient: _renew_subs 2024-03-25T18:21:03.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.437+0000 7f983ad8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:21:03.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.437+0000 7f983ad8d640 1 -- 172.21.15.64:0/640232983 --> v1:172.21.15.64:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9834139d20 con 0x7f98340a98f0 2024-03-25T18:21:03.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.437+0000 7f983ad8d640 10 client.4742.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:03.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.437+0000 7f983ad8d640 10 monclient: _renew_subs 2024-03-25T18:21:03.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.437+0000 7f983ad8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:21:03.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.437+0000 7f983ad8d640 1 -- 172.21.15.64:0/640232983 --> v1:172.21.15.64:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f983406f200 con 0x7f98340a98f0 2024-03-25T18:21:03.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.437+0000 7f983ad8d640 1 librados: init done 2024-03-25T18:21:03.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.438+0000 7f983ad8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:03.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.438+0000 7f983ad8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:03.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.438+0000 7f983ad8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:21:03.441 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.438+0000 7f983ad8d640 1 -- 172.21.15.64:0/640232983 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98341dc7e0 con 0x7f98340a98f0 2024-03-25T18:21:03.464 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.462+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 <== mon.2 v1:172.21.15.64:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9828003610 con 0x7f98340a98f0 2024-03-25T18:21:03.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.463+0000 7f98317fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:03.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.463+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 <== mon.2 v1:172.21.15.64:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1282490941 0 0) 0x7f9828003880 con 0x7f98340a98f0 2024-03-25T18:21:03.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.463+0000 7f98317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.463+0000 7f98317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:03.465 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.463+0000 7f98317fa640 10 monclient: dump: 2024-03-25T18:21:03.465 INFO:teuthology.misc.health.smithi064.stderr:epoch 1 2024-03-25T18:21:03.465 INFO:teuthology.misc.health.smithi064.stderr:fsid f3bd93ea-5bde-4b52-b027-4fe9ef2c973e 2024-03-25T18:21:03.465 INFO:teuthology.misc.health.smithi064.stderr:last_changed 2024-03-25T18:08:43.934429+0000 2024-03-25T18:21:03.465 INFO:teuthology.misc.health.smithi064.stderr:created 2024-03-25T18:08:43.934429+0000 2024-03-25T18:21:03.465 INFO:teuthology.misc.health.smithi064.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.465 INFO:teuthology.misc.health.smithi064.stderr:election_strategy: 3 2024-03-25T18:21:03.465 INFO:teuthology.misc.health.smithi064.stderr:0: v1:172.21.15.64:6789/0 mon.a 2024-03-25T18:21:03.465 INFO:teuthology.misc.health.smithi064.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-25T18:21:03.465 INFO:teuthology.misc.health.smithi064.stderr:2: v1:172.21.15.64:6790/0 mon.c 2024-03-25T18:21:03.465 INFO:teuthology.misc.health.smithi064.stderr: 2024-03-25T18:21:03.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.463+0000 7f9832ffd640 4 set_mon_vals no callback set 2024-03-25T18:21:03.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.463+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 <== mon.2 v1:172.21.15.64:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1866873509 0 0) 0x7f9828003aa0 con 0x7f98340a98f0 2024-03-25T18:21:03.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.464+0000 7f9833fff640 10 monclient: get_auth_request con 0x7f981806df70 auth_method 0 2024-03-25T18:21:03.466 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.464+0000 7f98317fa640 10 client.4742.objecter ms_handle_connect 0x7f981806df70 2024-03-25T18:21:03.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.467+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 <== mon.2 v1:172.21.15.64:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4227240091 0 0) 0x7f9828054680 con 0x7f98340a98f0 2024-03-25T18:21:03.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.467+0000 7f98317fa640 10 client.4742.objecter ms_dispatch 0x7f9834000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:21:03.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.467+0000 7f98317fa640 3 client.4742.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:21:03.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.467+0000 7f98317fa640 3 client.4742.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:21:03.469 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.467+0000 7f98317fa640 20 client.4742.objecter dump_active .. 0 homeless 2024-03-25T18:21:03.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.750+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 <== mon.2 v1:172.21.15.64:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9828050190 con 0x7f98340a98f0 2024-03-25T18:21:03.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.750+0000 7f98317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:03.752 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.750+0000 7f98317fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:03.893 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:47.006 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:03.940 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:47.052 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:03.940 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:11:47.018 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:03.942 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:11:47.020 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:03.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.968+0000 7f983ad8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:03.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.968+0000 7f983ad8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:03.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.968+0000 7f983ad8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.64:6790/0 2024-03-25T18:21:03.970 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:03.968+0000 7f983ad8d640 1 -- 172.21.15.64:0/640232983 --> v1:172.21.15.64:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9834074cc0 con 0x7f98340a98f0 2024-03-25T18:21:03.975 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:11:47.053 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:04.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:04.003+0000 7f98317fa640 1 -- 172.21.15.64:0/640232983 <== mon.2 v1:172.21.15.64:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 991200065) 0x7f98280520c0 con 0x7f98340a98f0 2024-03-25T18:21:04.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:04.003+0000 7f98317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:04.005 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:04.003+0000 7f98317fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:04.005 INFO:teuthology.misc.health.smithi064.stdout: 2024-03-25T18:21:04.005 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":11,"fsid":"f3bd93ea-5bde-4b52-b027-4fe9ef2c973e","created":"2024-03-25T18:09:33.453855+0000","modified":"2024-03-25T18:19:38.421162+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:38.421162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b7d57e83-d493-43bf-8e30-09bcea0e9df8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"739d7438-1775-493c-9156-4f5411c63dd5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ca83a5b9-4e42-4f79-907f-452b287a3324","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"315a607b-7263-41dc-8587-bd31883b50d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"95fe487e-5eef-4187-86b8-182e953b95c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b99f705f-8c5b-4b56-9e68-e6308b38ac09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c25d147-dd1a-4c2d-ac6f-d54ca65fee0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4e25c52d-8e4e-4bb5-b982-d962bae4c3a7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:04.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:04.005+0000 7f983ad8d640 10 librados: watch_flush enter 2024-03-25T18:21:04.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:04.005+0000 7f983ad8d640 10 librados: watch_flush exit 2024-03-25T18:21:04.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:04.005+0000 7f983ad8d640 20 client.4742.objecter shutdown clearing up homeless session... 2024-03-25T18:21:04.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:04.005+0000 7f983ad8d640 10 client.4742.objecter successfully canceled tick 2024-03-25T18:21:04.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:04.006+0000 7f983ad8d640 1 -- 172.21.15.64:0/640232983 >> v1:172.21.15.64:6800/2915543638 conn(0x7f981806df70 legacy=0x7f9818070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:04.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:04.006+0000 7f983ad8d640 10 monclient: shutdown 2024-03-25T18:21:04.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:04.006+0000 7f983ad8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:04.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:04.006+0000 7f983ad8d640 1 -- 172.21.15.64:0/640232983 >> v1:172.21.15.64:6790/0 conn(0x7f98340a98f0 legacy=0x7f98341dbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:04.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:04.006+0000 7f983ad8d640 1 -- 172.21.15.64:0/640232983 shutdown_connections 2024-03-25T18:21:04.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:04.006+0000 7f983ad8d640 1 -- 172.21.15.64:0/640232983 >> 172.21.15.64:0/640232983 conn(0x7f98340b7ea0 msgr2=0x7f9834074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:04.008 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:04.006+0000 7f983ad8d640 1 -- 172.21.15.64:0/640232983 shutdown_connections 2024-03-25T18:21:04.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:04.006+0000 7f983ad8d640 1 -- 172.21.15.64:0/640232983 wait complete. 2024-03-25T18:21:04.009 INFO:teuthology.misc.health.smithi064.stderr:2024-03-25T18:21:04.006+0000 7f983ad8d640 1 librados: shutdown 2024-03-25T18:21:04.010 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:11:47.122 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:04.013 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:47.126 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:04.018 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:04.018 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:21:04.019 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:21:04.020 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:21:04.020 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:21:04.025 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:47.138 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:04.058 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:11:47.170 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:06.044 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:11:26.091 35536== 2024-03-25T18:21:06.044 INFO:tasks.ceph.osd.0.smithi064.stderr:==00:00:11:26.092 35536== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:21:06.083 INFO:tasks.ceph.osd.0.smithi064.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:21:10.120 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:21:10.121 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:21:10.121 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:21:10.122 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:21:10.122 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:21:12.004 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:11:32.048 35537== 2024-03-25T18:21:12.004 INFO:tasks.ceph.osd.1.smithi064.stderr:==00:00:11:32.048 35537== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:21:12.037 INFO:tasks.ceph.osd.1.smithi064.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:21:16.224 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:21:16.224 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:21:16.224 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:21:16.224 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:21:16.225 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:21:18.002 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:11:38.041 35538== 2024-03-25T18:21:18.002 INFO:tasks.ceph.osd.2.smithi064.stderr:==00:00:11:38.041 35538== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:21:18.039 INFO:tasks.ceph.osd.2.smithi064.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:21:22.326 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:21:22.326 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:21:22.327 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:21:22.327 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:21:22.327 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:21:23.911 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:12:07.024 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:23.919 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:12:07.031 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:23.940 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:12:07.052 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:23.973 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:12:07.052 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:23.977 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:12:07.056 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:23.979 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:12:07.091 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:23.982 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:12:07.095 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:23.995 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:12:07.073 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:24.003 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:12:07.115 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:24.164 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:11:44.218 35535== 2024-03-25T18:21:24.164 INFO:tasks.ceph.osd.3.smithi064.stderr:==00:00:11:44.218 35535== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:21:24.205 INFO:tasks.ceph.osd.3.smithi064.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:21:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:21:28.429 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:21:28.430 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:21:28.430 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:21:28.430 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:21:30.303 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:11:50.358 35763== 2024-03-25T18:21:30.303 INFO:tasks.ceph.osd.4.smithi086.stderr:==00:00:11:50.358 35763== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:21:30.341 INFO:tasks.ceph.osd.4.smithi086.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:21:34.532 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:21:34.533 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:21:34.533 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:21:34.533 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:21:34.534 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:21:36.484 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:11:56.539 35762== 2024-03-25T18:21:36.484 INFO:tasks.ceph.osd.5.smithi086.stderr:==00:00:11:56.539 35762== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:21:36.521 INFO:tasks.ceph.osd.5.smithi086.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:21:40.635 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:21:40.635 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:21:40.636 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:21:40.636 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:21:40.636 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:21:42.289 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:12:02.340 35765== 2024-03-25T18:21:42.289 INFO:tasks.ceph.osd.6.smithi086.stderr:==00:00:12:02.340 35765== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:21:42.331 INFO:tasks.ceph.osd.6.smithi086.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:21:44.007 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:12:27.120 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:44.011 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:12:27.124 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:44.037 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:12:27.149 35170== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:44.057 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:12:27.135 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:44.059 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:12:27.138 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:44.063 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:12:27.175 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:44.070 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:12:27.182 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:44.073 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:12:27.152 35218== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:44.084 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:12:27.197 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:46.738 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:21:46.738 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:21:46.739 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:21:46.739 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:21:46.739 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:21:48.161 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:12:08.214 35764== 2024-03-25T18:21:48.161 INFO:tasks.ceph.osd.7.smithi086.stderr:==00:00:12:08.214 35764== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:21:48.197 INFO:tasks.ceph.osd.7.smithi086.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:21:52.840 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:21:52.840 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:21:52.841 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:21:52.841 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:21:52.841 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:21:52.841 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:21:52.843 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:21:52.842+0000 7fb774c06640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 35175) UID: 0 2024-03-25T18:21:52.843 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-03-25T18:21:52.842+0000 7fb774c06640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:21:52.869 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:21:52.870 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:21:52.870 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:21:52.895 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:21:52.895 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:21:52.895 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:21:52.895 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:21:52.925 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-03-25T18:21:52.913+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35135) UID: 0 2024-03-25T18:21:52.925 INFO:tasks.ceph.mon.a.smithi064.stderr:2024-03-25T18:21:52.915+0000 9f86640 -1 mon.a@0(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:21:53.477 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:12:36.589 35166== 2024-03-25T18:21:53.477 INFO:tasks.ceph.mon.a.smithi064.stderr:==00:00:12:36.589 35166== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:21:53.486 INFO:tasks.ceph.mon.a.smithi064.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:21:58.997 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:21:58.997 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:21:58.998 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:21:58.998 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:21:58.998 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:21:59.015 INFO:tasks.ceph.mon.c.smithi064.stderr:2024-03-25T18:21:59.001+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35137) UID: 0 2024-03-25T18:21:59.015 INFO:tasks.ceph.mon.c.smithi064.stderr:2024-03-25T18:21:59.002+0000 9f86640 -1 mon.c@2(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:21:59.565 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:12:42.678 35170== 2024-03-25T18:21:59.565 INFO:tasks.ceph.mon.c.smithi064.stderr:==00:00:12:42.678 35170== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:21:59.575 INFO:tasks.ceph.mon.c.smithi064.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:22:05.100 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:22:05.100 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:22:05.101 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:22:05.101 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:22:05.101 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:22:05.118 INFO:tasks.ceph.mon.b.smithi086.stderr:2024-03-25T18:22:05.104+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 35214) UID: 0 2024-03-25T18:22:05.118 INFO:tasks.ceph.mon.b.smithi086.stderr:2024-03-25T18:22:05.106+0000 9f86640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:22:05.772 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:12:48.850 35218== 2024-03-25T18:22:05.772 INFO:tasks.ceph.mon.b.smithi086.stderr:==00:00:12:48.850 35218== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:22:05.781 INFO:tasks.ceph.mon.b.smithi086.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:22:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:22:11.203 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:22:11.203 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:22:11.204 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:22:11.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:22:11.254 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-25T18:22:11.254 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:22:11.351 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-25T18:22:11.352 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:22:11.449 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-25T18:22:11.449 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:22:11.524 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi064.front.sepia.ceph.com 2024-03-25T18:22:11.524 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:22:11.608 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-25T18:22:11.608 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:22:11.747 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-25T18:22:11.747 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:22:11.825 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-25T18:22:11.825 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:22:11.884 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-25T18:22:11.885 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:22:11.943 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:22:11.944 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621263/data/mon.a.tgz 2024-03-25T18:22:11.945 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-03-25T18:22:11.961 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.zpxhDsDSTt 2024-03-25T18:22:11.961 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.zpxhDsDSTt 2024-03-25T18:22:12.060 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.zpxhDsDSTt 2024-03-25T18:22:12.146 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.zpxhDsDSTt is 134KB 2024-03-25T18:22:12.198 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.zpxhDsDSTt 2024-03-25T18:22:12.212 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621263/data/mon.c.tgz 2024-03-25T18:22:12.213 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-03-25T18:22:12.266 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.GYMzL7WVma 2024-03-25T18:22:12.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.GYMzL7WVma 2024-03-25T18:22:12.355 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.GYMzL7WVma 2024-03-25T18:22:12.439 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.GYMzL7WVma is 134KB 2024-03-25T18:22:12.461 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.GYMzL7WVma 2024-03-25T18:22:12.474 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621263/data/mon.b.tgz 2024-03-25T18:22:12.474 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-03-25T18:22:12.492 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.rH2WUktcv1 2024-03-25T18:22:12.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.rH2WUktcv1 2024-03-25T18:22:12.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.rH2WUktcv1 2024-03-25T18:22:12.686 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.rH2WUktcv1 is 134KB 2024-03-25T18:22:12.739 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.rH2WUktcv1 2024-03-25T18:22:12.754 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:22:12.754 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:22:12.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:22:13.111 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:22:13.330 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:22:13.332 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621263/remote/smithi064/crash 2024-03-25T18:22:13.333 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:22:13.359 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621263/remote/smithi086/crash 2024-03-25T18:22:13.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:22:13.388 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:22:13.388 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:22:13.404 DEBUG:teuthology.orchestra.run.smithi086:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:22:13.428 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:22:13.428 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41165.log 2024-03-25T18:22:13.428 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42390.log 2024-03-25T18:22:13.428 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40892.log 2024-03-25T18:22:13.475 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:22:13.475 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:22:13.475 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log/var/log/ceph/ceph-osd.6.log: 2024-03-25T18:22:13.475 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:22:13.475 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/valgrind/osd.5.log: 72.1% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:22:13.475 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:22:13.475 INFO:teuthology.orchestra.run.smithi086.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:22:13.475 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:22:13.475 INFO:teuthology.orchestra.run.smithi086.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:22:13.475 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35412.log 2024-03-25T18:22:13.475 INFO:teuthology.orchestra.run.smithi086.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:22:13.476 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35532.log 2024-03-25T18:22:13.476 INFO:teuthology.orchestra.run.smithi086.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:22:13.476 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35412.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:22:13.476 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35532.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:22:13.476 INFO:teuthology.orchestra.run.smithi086.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35412.log.gz 2024-03-25T18:22:13.476 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35614.log 2024-03-25T18:22:13.476 INFO:teuthology.orchestra.run.smithi086.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35532.log.gz 2024-03-25T18:22:13.476 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.4.log: 88.6%gzip -- replaced with /var/log/ceph/ceph.audit.log.gz -5 2024-03-25T18:22:13.476 INFO:teuthology.orchestra.run.smithi086.stderr: --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:22:13.476 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35614.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:22:13.476 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35452.log 2024-03-25T18:22:13.476 INFO:teuthology.orchestra.run.smithi086.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35614.log.gz 2024-03-25T18:22:13.476 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:22:13.476 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35572.log 2024-03-25T18:22:13.476 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-mgr.x.log: gzip 87.6% -5 --verbose -- /var/log/ceph/ceph-client.admin.35663.log 2024-03-25T18:22:13.476 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35452.log.gz 2024-03-25T18:22:13.476 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35492.log 2024-03-25T18:22:13.477 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35663.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:22:13.477 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35572.log.gz 2024-03-25T18:22:13.477 INFO:teuthology.orchestra.run.smithi086.stderr: 87.6%/var/log/ceph/ceph-client.admin.35492.log: -- replaced with /var/log/ceph/ceph-client.admin.35663.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35372.log 2024-03-25T18:22:13.477 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:22:13.477 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35492.log.gz 2024-03-25T18:22:13.477 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:22:13.477 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35372.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:22:13.477 INFO:teuthology.orchestra.run.smithi086.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35372.log.gz 2024-03-25T18:22:13.478 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41984.log 2024-03-25T18:22:13.478 INFO:teuthology.orchestra.run.smithi064.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41165.log.gz 2024-03-25T18:22:13.478 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40229.log 2024-03-25T18:22:13.478 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40892.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42390.log.gz 2024-03-25T18:22:13.478 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41789.log 2024-03-25T18:22:13.478 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40307.log 2024-03-25T18:22:13.478 INFO:teuthology.orchestra.run.smithi064.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40892.log.gz 2024-03-25T18:22:13.478 INFO:teuthology.orchestra.run.smithi064.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41984.log.gz 2024-03-25T18:22:13.478 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41243.log 2024-03-25T18:22:13.478 INFO:teuthology.orchestra.run.smithi064.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40229.log.gz 2024-03-25T18:22:13.478 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40307.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41789.log.gz 2024-03-25T18:22:13.479 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41399.log 2024-03-25T18:22:13.479 INFO:teuthology.orchestra.run.smithi064.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40307.log.gz 2024-03-25T18:22:13.479 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41009.log 2024-03-25T18:22:13.479 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41243.log: /var/log/ceph/ceph-client.admin.41399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39135.log 2024-03-25T18:22:13.479 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41009.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:22:13.479 INFO:teuthology.orchestra.run.smithi064.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41243.log.gz 2024-03-25T18:22:13.479 INFO:teuthology.orchestra.run.smithi064.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41399.log.gz 2024-03-25T18:22:13.479 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39135.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:22:13.479 INFO:teuthology.orchestra.run.smithi064.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41009.log.gz 2024-03-25T18:22:13.479 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:22:13.479 INFO:teuthology.orchestra.run.smithi064.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39135.log.gz 2024-03-25T18:22:13.479 INFO:teuthology.orchestra.run.smithi064.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:22:13.479 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:22:13.479 INFO:teuthology.orchestra.run.smithi064.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:22:13.479 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:22:13.479 INFO:teuthology.orchestra.run.smithi064.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:22:13.480 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:22:13.480 INFO:teuthology.orchestra.run.smithi064.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:22:13.480 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39566.log 2024-03-25T18:22:13.480 INFO:teuthology.orchestra.run.smithi064.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:22:13.480 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41867.log 2024-03-25T18:22:13.480 INFO:teuthology.orchestra.run.smithi064.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:22:13.480 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39722.log 2024-03-25T18:22:13.480 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39917.log 2024-03-25T18:22:13.480 INFO:teuthology.orchestra.run.smithi064.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39566.log.gz 2024-03-25T18:22:13.480 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40385.log 2024-03-25T18:22:13.480 INFO:teuthology.orchestra.run.smithi064.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41867.log.gz 2024-03-25T18:22:13.480 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39878.log 2024-03-25T18:22:13.480 INFO:teuthology.orchestra.run.smithi064.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39722.log.gz 2024-03-25T18:22:13.480 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40385.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39917.log.gz 2024-03-25T18:22:13.480 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39605.log 2024-03-25T18:22:13.480 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39254.log 2024-03-25T18:22:13.480 INFO:teuthology.orchestra.run.smithi064.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40385.log.gz 2024-03-25T18:22:13.481 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41360.log 2024-03-25T18:22:13.481 INFO:teuthology.orchestra.run.smithi064.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39878.log.gz 2024-03-25T18:22:13.481 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39254.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39605.log.gz 2024-03-25T18:22:13.481 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40619.log 2024-03-25T18:22:13.481 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35534.log 2024-03-25T18:22:13.481 INFO:teuthology.orchestra.run.smithi064.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39254.log.gz 2024-03-25T18:22:13.481 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40424.log 2024-03-25T18:22:13.481 INFO:teuthology.orchestra.run.smithi064.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41360.log.gz 2024-03-25T18:22:13.481 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35534.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40619.log.gz 2024-03-25T18:22:13.481 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39332.log 2024-03-25T18:22:13.481 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38880.log 2024-03-25T18:22:13.481 INFO:teuthology.orchestra.run.smithi064.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.35534.log.gz 2024-03-25T18:22:13.481 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41906.log 2024-03-25T18:22:13.481 INFO:teuthology.orchestra.run.smithi064.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40424.log.gz 2024-03-25T18:22:13.481 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38880.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39332.log.gz 2024-03-25T18:22:13.481 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40736.log 2024-03-25T18:22:13.481 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39683.log 2024-03-25T18:22:13.482 INFO:teuthology.orchestra.run.smithi064.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38880.log.gz 2024-03-25T18:22:13.482 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40931.log 2024-03-25T18:22:13.482 INFO:teuthology.orchestra.run.smithi064.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41906.log.gz 2024-03-25T18:22:13.482 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39683.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40736.log.gz 2024-03-25T18:22:13.482 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40346.log 2024-03-25T18:22:13.482 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39449.log 2024-03-25T18:22:13.482 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40931.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39683.log.gz 2024-03-25T18:22:13.482 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41633.log 2024-03-25T18:22:13.482 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40541.log 2024-03-25T18:22:13.482 INFO:teuthology.orchestra.run.smithi064.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40931.log.gz 2024-03-25T18:22:13.482 INFO:teuthology.orchestra.run.smithi064.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40346.log.gz 2024-03-25T18:22:13.482 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40580.log 2024-03-25T18:22:13.482 INFO:teuthology.orchestra.run.smithi064.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39449.log.gz 2024-03-25T18:22:13.482 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39761.log 2024-03-25T18:22:13.482 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40541.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41633.log.gz 2024-03-25T18:22:13.482 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40580.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:22:13.482 INFO:teuthology.orchestra.run.smithi064.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40541.log.gz 2024-03-25T18:22:13.483 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39410.log 2024-03-25T18:22:13.483 INFO:teuthology.orchestra.run.smithi064.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40580.log.gz 2024-03-25T18:22:13.483 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- 87.0% /var/log/ceph/ceph-client.admin.37693.log -- replaced with /var/log/ceph/ceph-client.admin.39761.log.gz 2024-03-25T18:22:13.483 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-25T18:22:13.483 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41204.log 2024-03-25T18:22:13.483 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39410.log: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:22:13.483 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39293.log 2024-03-25T18:22:13.483 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41204.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39410.log.gz 2024-03-25T18:22:13.483 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39800.log 2024-03-25T18:22:13.483 INFO:teuthology.orchestra.run.smithi064.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37693.log.gz 2024-03-25T18:22:13.483 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39371.log 2024-03-25T18:22:13.483 INFO:teuthology.orchestra.run.smithi064.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41204.log.gz 2024-03-25T18:22:13.483 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42102.log 2024-03-25T18:22:13.483 INFO:teuthology.orchestra.run.smithi064.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39293.log.gz 2024-03-25T18:22:13.483 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40034.log 2024-03-25T18:22:13.484 INFO:teuthology.orchestra.run.smithi064.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39800.log.gz 2024-03-25T18:22:13.484 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39644.log 2024-03-25T18:22:13.484 INFO:teuthology.orchestra.run.smithi064.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39371.log.gz 2024-03-25T18:22:13.484 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40034.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:22:13.484 INFO:teuthology.orchestra.run.smithi064.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42102.log.gz 2024-03-25T18:22:13.484 INFO:teuthology.orchestra.run.smithi064.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40034.log.gz 2024-03-25T18:22:13.484 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42234.log 2024-03-25T18:22:13.484 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41087.log 2024-03-25T18:22:13.484 INFO:teuthology.orchestra.run.smithi064.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39644.log.gz 2024-03-25T18:22:13.484 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40970.log 2024-03-25T18:22:13.484 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41087.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42234.log.gz 2024-03-25T18:22:13.484 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40112.log 2024-03-25T18:22:13.484 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41048.log 2024-03-25T18:22:13.484 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40970.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41087.log.gz 2024-03-25T18:22:13.485 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42062.log 2024-03-25T18:22:13.485 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41048.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40970.log.gz 2024-03-25T18:22:13.485 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41711.log 2024-03-25T18:22:13.485 INFO:teuthology.orchestra.run.smithi064.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40112.log.gz 2024-03-25T18:22:13.485 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42062.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41048.log.gz 2024-03-25T18:22:13.485 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39956.log 2024-03-25T18:22:13.485 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41711.log: -- /var/log/ceph/ceph-client.admin.35732.log 2024-03-25T18:22:13.485 INFO:teuthology.orchestra.run.smithi064.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42062.log.gz 2024-03-25T18:22:13.485 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42273.log 2024-03-25T18:22:13.485 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35732.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41711.log.gz 2024-03-25T18:22:13.485 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:22:13.485 INFO:teuthology.orchestra.run.smithi064.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39956.log.gz 2024-03-25T18:22:13.485 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39488.log 2024-03-25T18:22:13.485 INFO:teuthology.orchestra.run.smithi064.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35732.log.gz 2024-03-25T18:22:13.485 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40190.log 2024-03-25T18:22:13.486 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39488.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42273.log.gz 2024-03-25T18:22:13.486 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40775.log 2024-03-25T18:22:13.486 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40268.log 2024-03-25T18:22:13.486 INFO:teuthology.orchestra.run.smithi064.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39488.log.gz 2024-03-25T18:22:13.486 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40775.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29325.log 2024-03-25T18:22:13.486 INFO:teuthology.orchestra.run.smithi064.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40190.log.gz 2024-03-25T18:22:13.486 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40151.log 2024-03-25T18:22:13.486 INFO:teuthology.orchestra.run.smithi064.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40775.log.gz 2024-03-25T18:22:13.486 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.tmp-client.admin.29325.log: gzip 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29325.log.gz -5 2024-03-25T18:22:13.486 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42023.log 2024-03-25T18:22:13.486 INFO:teuthology.orchestra.run.smithi064.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40268.log.gz 2024-03-25T18:22:13.486 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40073.log 2024-03-25T18:22:13.486 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41438.log 2024-03-25T18:22:13.486 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42023.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40151.log.gz 2024-03-25T18:22:13.487 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40073.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:22:13.487 INFO:teuthology.orchestra.run.smithi064.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42023.log.gz 2024-03-25T18:22:13.487 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41750.log 2024-03-25T18:22:13.487 INFO:teuthology.orchestra.run.smithi064.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40073.log.gz 2024-03-25T18:22:13.487 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41477.log 2024-03-25T18:22:13.487 INFO:teuthology.orchestra.run.smithi064.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41438.log.gz 2024-03-25T18:22:13.487 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35585.log 2024-03-25T18:22:13.487 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41477.log: gzip -5 --verbose -- 85.8% /var/log/ceph/ceph-client.admin.42143.log 2024-03-25T18:22:13.487 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41750.log.gz 2024-03-25T18:22:13.487 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35585.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41477.log.gz 2024-03-25T18:22:13.487 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40697.log 2024-03-25T18:22:13.487 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42143.log: gzip 85.6% -- replaced with /var/log/ceph/ceph-client.admin.35585.log.gz 2024-03-25T18:22:13.487 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41126.log 2024-03-25T18:22:13.487 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40697.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42143.log.gz 2024-03-25T18:22:13.487 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39995.log 2024-03-25T18:22:13.487 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41126.log: gzip -5 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40697.log.gz 2024-03-25T18:22:13.488 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35177.log 2024-03-25T18:22:13.488 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39995.log: gzip 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41126.log.gz 2024-03-25T18:22:13.488 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41555.log 2024-03-25T18:22:13.488 INFO:teuthology.orchestra.run.smithi064.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39995.log.gz 2024-03-25T18:22:13.488 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.35177.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:22:13.488 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/ceph-client.admin.41555.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41282.log 2024-03-25T18:22:13.488 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.0.log: gzip 90.1% -- replaced with /var/log/ceph/ceph-client.admin.35177.log.gz 2024-03-25T18:22:13.488 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41594.log 2024-03-25T18:22:13.488 INFO:teuthology.orchestra.run.smithi064.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41555.log.gz 2024-03-25T18:22:13.488 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41282.log: -- /var/log/ceph/ceph-client.admin.39527.log 2024-03-25T18:22:13.488 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41594.log: gzip -5 --verbose 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41282.log.gz 2024-03-25T18:22:13.488 INFO:teuthology.orchestra.run.smithi064.stderr: -- /var/log/ceph/ceph-client.admin.40502.log 2024-03-25T18:22:13.488 INFO:teuthology.orchestra.run.smithi064.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41594.log.gz 2024-03-25T18:22:13.488 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41516.log 2024-03-25T18:22:13.488 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39527.log: /var/log/ceph/ceph-client.admin.40502.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:22:13.488 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41516.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40502.log.gz 2024-03-25T18:22:13.489 INFO:teuthology.orchestra.run.smithi064.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39527.log.gz 2024-03-25T18:22:13.489 INFO:teuthology.orchestra.run.smithi064.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41516.log.gz 2024-03-25T18:22:13.489 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41828.log 2024-03-25T18:22:13.489 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:22:13.489 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42194.log 2024-03-25T18:22:13.489 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.2.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41828.log.gz 2024-03-25T18:22:13.489 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42351.log 2024-03-25T18:22:13.489 INFO:teuthology.orchestra.run.smithi064.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:22:13.489 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41321.log 2024-03-25T18:22:13.489 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42351.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42194.log.gz 2024-03-25T18:22:13.489 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42351.log.gz 2024-03-25T18:22:13.489 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41945.log 2024-03-25T18:22:13.489 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42312.log 2024-03-25T18:22:13.489 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41945.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41321.log.gz 2024-03-25T18:22:13.489 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36604.log 2024-03-25T18:22:13.489 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42312.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41945.log.gz 2024-03-25T18:22:13.490 INFO:teuthology.orchestra.run.smithi064.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42312.log.gz 2024-03-25T18:22:13.490 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41672.log 2024-03-25T18:22:13.490 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.36604.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36604.log.gz 2024-03-25T18:22:13.490 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40814.log 2024-03-25T18:22:13.490 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39176.log 2024-03-25T18:22:13.490 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41672.log: /var/log/ceph/ceph-client.admin.40814.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41672.log.gz 2024-03-25T18:22:13.490 INFO:teuthology.orchestra.run.smithi064.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40814.log.gz 2024-03-25T18:22:13.490 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39839.log 2024-03-25T18:22:13.490 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39176.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39176.log.gz 2024-03-25T18:22:13.490 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39215.log 2024-03-25T18:22:13.490 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40463.log 2024-03-25T18:22:13.490 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39215.log: /var/log/ceph/ceph-client.admin.39839.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39215.log.gz 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39839.log.gz 2024-03-25T18:22:13.490 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-25T18:22:13.490 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40853.log 2024-03-25T18:22:13.490 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40658.log 2024-03-25T18:22:13.491 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40463.log: /var/log/ceph/ceph-client.admin.40853.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40463.log.gz 2024-03-25T18:22:13.491 INFO:teuthology.orchestra.run.smithi064.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40853.log.gz 2024-03-25T18:22:13.491 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39096.log 2024-03-25T18:22:13.491 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40658.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40658.log.gz 2024-03-25T18:22:13.491 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:22:13.491 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39096.log: /var/log/ceph/ceph-mon.c.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39096.log.gz 2024-03-25T18:22:13.491 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-mon.b.log: 93.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:22:13.530 INFO:teuthology.orchestra.run.smithi064.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:22:14.296 INFO:teuthology.orchestra.run.smithi064.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:22:14.322 INFO:teuthology.orchestra.run.smithi086.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:22:14.376 INFO:teuthology.orchestra.run.smithi086.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:22:14.566 INFO:teuthology.orchestra.run.smithi064.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:22:14.638 INFO:teuthology.orchestra.run.smithi086.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:22:14.656 INFO:teuthology.orchestra.run.smithi064.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:22:14.662 INFO:teuthology.orchestra.run.smithi086.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:22:14.675 INFO:teuthology.orchestra.run.smithi064.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:22:18.371 INFO:teuthology.orchestra.run.smithi064.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:22:18.554 INFO:teuthology.orchestra.run.smithi086.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:22:18.556 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-25T18:22:18.556 INFO:teuthology.orchestra.run.smithi086.stderr:real 0m5.106s 2024-03-25T18:22:18.556 INFO:teuthology.orchestra.run.smithi086.stderr:user 0m8.931s 2024-03-25T18:22:18.556 INFO:teuthology.orchestra.run.smithi086.stderr:sys 0m0.332s 2024-03-25T18:22:18.863 INFO:teuthology.orchestra.run.smithi064.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:22:18.865 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-25T18:22:18.865 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m5.449s 2024-03-25T18:22:18.865 INFO:teuthology.orchestra.run.smithi064.stderr:user 0m14.417s 2024-03-25T18:22:18.865 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m0.505s 2024-03-25T18:22:18.865 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:22:18.866 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621263/remote/smithi064/log 2024-03-25T18:22:18.867 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:22:21.145 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621263/remote/smithi086/log 2024-03-25T18:22:21.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:22:22.604 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:22:22.788 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=25ca1ad17ed24586838005aa2e26c1a6 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:22:22.791 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:22:22.800 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:22:22.815 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:22:22.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:22:22.821 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:22:22.903 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:22:22.911 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:22:22.911 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:22:22.913 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:22:22.927 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:22:22.929 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:22:22.932 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:22:22.933 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-03-25T18:22:22.933 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 52 +577us[ +577us] +/- 96ms 2024-03-25T18:22:22.933 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv02.front.sepia.ceph.com 3 6 377 53 -1300us[-1309us] +/- 78ms 2024-03-25T18:22:22.933 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 +745us[ +745us] +/- 67ms 2024-03-25T18:22:22.933 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:22:22.934 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:22:22.934 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-03-25T18:22:22.934 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 54 +563us[ +563us] +/- 96ms 2024-03-25T18:22:22.934 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv02.front.sepia.ceph.com 3 6 377 54 -1301us[-1300us] +/- 78ms 2024-03-25T18:22:22.934 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 +737us[ +737us] +/- 67ms 2024-03-25T18:22:22.934 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:22:22.935 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:22:22.943 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:22:22.944 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:22:22.951 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:22:23.045 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:22:23.116 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:22:23.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:22:23.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:22:23.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:22:23.179 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:22:23.188 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:22:23.195 INFO:teuthology.task.internal:Duration was 7373.664021 seconds 2024-03-25T18:22:23.196 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:22:23.205 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:22:23.205 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:22:23.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:22:23.250 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:22:23.263 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:22:23.521 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:22:23.521 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-03-25T18:22:23.521 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:22:23.541 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2024-03-25T18:22:23.541 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:22:23.564 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:22:23.564 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:22:23.583 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:22:23.632 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:22:23.632 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:22:23.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:22:23.775 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:22:23.784 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:22:23.785 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:22:23.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:22:23.838 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:22:23.847 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:22:23.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:22:23.876 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-03-25T18:22:23.903 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2024-03-25T18:22:23.939 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:22:23.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:22:23.959 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:22:23.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:22:23.993 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:22:24.020 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:22:24.021 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621263/remote/smithi064 2024-03-25T18:22:24.021 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:22:24.055 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621263/remote/smithi086 2024-03-25T18:22:24.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:22:24.090 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:22:24.090 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:22:24.093 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:22:24.140 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:22:24.150 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:22:24.151 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:22:24.160 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:22:24.160 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:22:24.162 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:22:24.174 INFO:teuthology.orchestra.run.smithi064.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:22 /home/ubuntu/cephtest 2024-03-25T18:22:24.194 INFO:teuthology.orchestra.run.smithi086.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:22 /home/ubuntu/cephtest 2024-03-25T18:22:24.195 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:22:24.204 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:22:24.227 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:22:24.253 DEBUG:teuthology.nuke:shortname: smithi064 2024-03-25T18:22:24.254 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:22:24.279 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621263', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:10:59.941756', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOsESA1elaXn55kueK3fg/Dl/CEXysJyvHdNcX1qTpJ6zrBpeeQqkkwPdMgV3RBpENoBbyHZLR0nrbBUrKKPduo='} 2024-03-25T18:22:24.282 DEBUG:teuthology.nuke:shortname: smithi086 2024-03-25T18:22:24.282 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:22:24.305 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621263', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:10:59.944183', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBh9+NaMT+DLhes+BYo8wW/byWZV2VoUHyX7ijMCUMoyWHexkOSxRdg1lbBDy8P4D9z1Hprw2BYNCbuEm1Uygsc='} 2024-03-25T18:22:24.326 INFO:teuthology.orchestra.console.smithi064:Power off 2024-03-25T18:22:24.326 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:22:24.350 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off 2024-03-25T18:22:24.350 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:22:24.366 INFO:teuthology.orchestra.console.smithi086:Power off 2024-03-25T18:22:24.367 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:22:24.375 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-25T18:22:24.388 DEBUG:teuthology.orchestra.console.smithi086:power off output: Chassis Power Control: Down/Off 2024-03-25T18:22:24.388 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:22:24.410 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-03-25T18:22:28.376 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:22:28.410 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:22:28.485 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-25T18:22:28.519 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-03-25T18:22:32.491 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:22:32.520 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:22:32.602 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-25T18:22:32.629 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-03-25T18:22:36.606 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:22:36.630 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:22:36.716 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2024-03-25T18:22:36.739 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is off 2024-03-25T18:22:36.817 INFO:teuthology.orchestra.console.smithi064:Power off completed 2024-03-25T18:22:36.840 INFO:teuthology.orchestra.console.smithi086:Power off completed 2024-03-25T18:22:36.944 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7373.664020776749 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=25ca1ad17ed24586838005aa2e26c1a6 status: fail success: false 2024-03-25T18:22:36.945 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:22:37.020 INFO:teuthology.run:FAIL